[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:42:48 dist-test-slave-7g3l krb5kdc[1982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:42:48 dist-test-slave-7g3l krb5kdc[1982](info): set up 2 sockets
Sep 27 13:42:48 dist-test-slave-7g3l krb5kdc[1982](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 13:42:49.277205 1975 mini_kdc.cc:121] Time spent starting KDC: real 2.038s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:42:49 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:42:49.437227 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46529
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:49.746456 1995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:49.746713 1995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:49.746796 1995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:49.754014 1995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:42:49.754099 1995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:49.754164 1995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:49.754211 1995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:42:49.754244 1995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:42:49.763319 1995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:49.764448 1995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:49.765605 1995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:49.777347 2000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:49.777380 2001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:49.777566 1995 server_base.cc:1034] running on GCE node
W20260927 13:42:49.777441 2003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:49.778535 1995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:49.779856 1995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:49.781070 1995 hybrid_clock.cc:648] HybridClock initialized: now 1790516569780994 us; error 71 us; skew 500 ppm
I20260927 13:42:49.781394 1995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:49 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:49.784224 1995 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:42:49.785504 1995 webserver.cc:460] Webserver started at http://127.1.237.254:43723/ using document root <none> and password file <none>
I20260927 13:42:49.786077 1995 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:49.786168 1995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:49.786394 1995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:49.789158 1995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "f2bae4e1f50d49a488c0912d971d6d19"
format_stamp: "Formatted at 2026-09-27 13:42:49 on dist-test-slave-7g3l"
server_key: "a370c02af7bc267893fd4b3f0c9361d5"
server_key_iv: "cd41884714cf5b4c53d3026ce9409cc3"
server_key_version: "encryptionkey@0"
I20260927 13:42:49.789824 1995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "f2bae4e1f50d49a488c0912d971d6d19"
format_stamp: "Formatted at 2026-09-27 13:42:49 on dist-test-slave-7g3l"
server_key: "a370c02af7bc267893fd4b3f0c9361d5"
server_key_iv: "cd41884714cf5b4c53d3026ce9409cc3"
server_key_version: "encryptionkey@0"
I20260927 13:42:49.795040 1995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:42:49.798938 2010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:49.800269 1995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:42:49.800401 1995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "f2bae4e1f50d49a488c0912d971d6d19"
format_stamp: "Formatted at 2026-09-27 13:42:49 on dist-test-slave-7g3l"
server_key: "a370c02af7bc267893fd4b3f0c9361d5"
server_key_iv: "cd41884714cf5b4c53d3026ce9409cc3"
server_key_version: "encryptionkey@0"
I20260927 13:42:49.800567 1995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:49.821353 1995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:49.824934 1995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:49.825557 1995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:49.843107 1995 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46529
I20260927 13:42:49.843127 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46529 every 8 connection(s)
I20260927 13:42:49.844357 1995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:42:49.847862 2062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:49.854270 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 1995
I20260927 13:42:49.854571 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:42:49.856375 1975 external_mini_cluster.cc:1428] Setting key 895aea00dd960c52b9d7611526b94bff
I20260927 13:42:49.857831 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19: Bootstrap starting.
I20260927 13:42:49.861523 2062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:49.862763 2062 log.cc:824] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:49.865165 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19: No bootstrap required, opened a new log
I20260927 13:42:49.873013 2062 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } }
I20260927 13:42:49.873377 2062 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:49.873503 2062 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2bae4e1f50d49a488c0912d971d6d19, State: Initialized, Role: FOLLOWER
I20260927 13:42:49.874114 2062 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } }
I20260927 13:42:49.874327 2062 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:42:49.874469 2062 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:42:49.874630 2062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:49.876660 2062 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } }
I20260927 13:42:49.877130 2062 leader_election.cc:302] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2bae4e1f50d49a488c0912d971d6d19; no voters:
I20260927 13:42:49.877791 2062 leader_election.cc:288] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:42:49.877920 2067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:49.879027 2067 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [term 1 LEADER]: Becoming Leader. State: Replica: f2bae4e1f50d49a488c0912d971d6d19, State: Running, Role: LEADER
I20260927 13:42:49.879537 2067 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } }
I20260927 13:42:49.879716 2062 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:42:49.882179 2069 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2bae4e1f50d49a488c0912d971d6d19. Latest consensus state: current_term: 1 leader_uuid: "f2bae4e1f50d49a488c0912d971d6d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } } }
I20260927 13:42:49.882501 2069 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [sys.catalog]: This master's current role is: LEADER
I20260927 13:42:49.885274 2072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:42:49.885113 2068 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2bae4e1f50d49a488c0912d971d6d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bae4e1f50d49a488c0912d971d6d19" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46529 } } }
I20260927 13:42:49.885483 2068 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19 [sys.catalog]: This master's current role is: LEADER
Sep 27 13:42:49 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:49.891408 2072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:42:49.901743 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:49.859786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.1.237.254:46529)
0927 13:42:49.860548 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:49.860558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:49.861720 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 13:42:49.872610 (+ 10890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:49.872629 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:49.872950 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:49.873323 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:49.876216 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:49.877655 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:49.880013 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:49.881552 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:49.881726 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:49.889690 (+ 7964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:49.889702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:49.889719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:49.889771 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:49.894382 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:49.895337 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:49.895344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:49.895368 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:49.895543 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:49.896053 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:49.896058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:49.896062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:49.896672 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:49.896972 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:49.899831 (+ 2859us) server_negotiation.cc:299] Negotiation successful
0927 13:42:49.900219 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":381,"threads_started":1}
I20260927 13:42:49.914288 2072 catalog_manager.cc:1347] Generated new cluster ID: 53fa6d4967be4056a408badef5d98665
I20260927 13:42:49.914394 2072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:42:49.926425 2072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:42:49.928155 2072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:42:49.937803 2072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2bae4e1f50d49a488c0912d971d6d19: Generated new TSK 0
I20260927 13:42:49.938575 2072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:42:49.982415 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:50.300699 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:50.300944 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:50.301126 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:50.308063 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:50.308187 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:50.308359 2088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:42:50.318356 2088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:50.319532 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:50.320709 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:50.333137 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:50.333146 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:50.333366 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:50.333760 2088 server_base.cc:1034] running on GCE node
I20260927 13:42:50.334331 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:50.335134 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:50.336403 2088 hybrid_clock.cc:648] HybridClock initialized: now 1790516570336363 us; error 29 us; skew 500 ppm
I20260927 13:42:50.336727 2088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:50 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:50.339648 2088 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:42:50.340801 2088 webserver.cc:460] Webserver started at http://127.1.237.193:41835/ using document root <none> and password file <none>
I20260927 13:42:50.341388 2088 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:50.341522 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:50.341818 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:50.344473 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "afff2bbaa4c74aad9a0fe70869df976d"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "d28e5f4da99550e4deab21515af312d7"
server_key_iv: "f4f86d40b4fcefd04479e88af5bd9a27"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.345161 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "afff2bbaa4c74aad9a0fe70869df976d"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "d28e5f4da99550e4deab21515af312d7"
server_key_iv: "f4f86d40b4fcefd04479e88af5bd9a27"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.350085 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 13:42:50.353550 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.354666 2088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:42:50.354820 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "afff2bbaa4c74aad9a0fe70869df976d"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "d28e5f4da99550e4deab21515af312d7"
server_key_iv: "f4f86d40b4fcefd04479e88af5bd9a27"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.354990 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:50.372818 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:50.376610 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:50.376876 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:50.377674 2088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:50.379240 2088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:50.379334 2088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.379472 2088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:50.379540 2088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.412222 2088 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37051
I20260927 13:42:50.412302 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37051 every 8 connection(s)
I20260927 13:42:50.413360 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:42:50.421124 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2088
I20260927 13:42:50.421329 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:42:50.421869 1975 external_mini_cluster.cc:1428] Setting key f8a4756783bf7acef4810b7b70d938fd
Sep 27 13:42:50 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:50.431905 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:50.415714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:47897 (local address 127.1.237.254:46529)
0927 13:42:50.416153 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:50.416157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:50.416941 (+ 784us) server_negotiation.cc:407] Connection header received
0927 13:42:50.418625 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:50.418630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:50.418712 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:50.418821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:50.420180 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.420868 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.422458 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.422913 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.423025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:50.426658 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:50.426662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:50.426665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:50.426693 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:50.429202 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:50.429846 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:50.429853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:50.429856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:50.429965 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:50.430347 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:50.430353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:50.430357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:50.430602 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:50.430742 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:50.431368 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 13:42:50.431557 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
I20260927 13:42:50.432767 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:50.416191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46529 (local address 127.1.237.193:47897)
0927 13:42:50.416801 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:50.416838 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:50.418375 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:50.419051 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:50.419058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:50.419547 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:50.420019 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:50.420034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.421009 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.421021 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:50.422311 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:50.422323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.423126 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.423131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:50.423387 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:50.424045 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:50.424078 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:50.426461 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:50.429330 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:50.429339 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:50.429352 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:50.429735 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:50.430074 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:50.430079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:50.430083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:50.430243 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:50.430719 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:50.430727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:50.431010 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 13:42:50.431300 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 13:42:50.431744 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":229,"threads_started":1}
I20260927 13:42:50.434152 2216 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46529
I20260927 13:42:50.434418 2216 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:50.435029 2216 heartbeater.cc:511] Master 127.1.237.254:46529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:42:50.437899 2027 ts_manager.cc:194] Registered new tserver with Master: afff2bbaa4c74aad9a0fe70869df976d (127.1.237.193:37051)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:42:50.440056 2027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:47897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:42:50.462194 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:50.769942 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:50.770208 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:50.770313 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:50.777246 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:50.777334 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:50.777432 2222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:42:50.786623 2222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:50.787854 2222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:50.789067 2222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:50.801296 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:50.801331 2228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:50.801447 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:50.802428 2222 server_base.cc:1034] running on GCE node
I20260927 13:42:50.803036 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:50.803949 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:50.805168 2222 hybrid_clock.cc:648] HybridClock initialized: now 1790516570805115 us; error 52 us; skew 500 ppm
I20260927 13:42:50.805440 2222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:50 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:50.808252 2222 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:42:50.809547 2222 webserver.cc:460] Webserver started at http://127.1.237.194:41483/ using document root <none> and password file <none>
I20260927 13:42:50.810161 2222 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:50.810297 2222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:50.810595 2222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:50.813360 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "90061e2659824ea78da2ea47abf64abf"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "9673f4fb30b08ae03b01b88bbed48879"
server_key_iv: "a8547ea98b9bcc7cd9b44ca42fcdc22d"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.814096 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "90061e2659824ea78da2ea47abf64abf"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "9673f4fb30b08ae03b01b88bbed48879"
server_key_iv: "a8547ea98b9bcc7cd9b44ca42fcdc22d"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.818764 2222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:42:50.822158 2237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.823307 2222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 13:42:50.823464 2222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "90061e2659824ea78da2ea47abf64abf"
format_stamp: "Formatted at 2026-09-27 13:42:50 on dist-test-slave-7g3l"
server_key: "9673f4fb30b08ae03b01b88bbed48879"
server_key_iv: "a8547ea98b9bcc7cd9b44ca42fcdc22d"
server_key_version: "encryptionkey@0"
I20260927 13:42:50.823642 2222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:50.848769 2222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:50.853417 2222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:50.853700 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:50.854609 2222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:50.856516 2222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:50.856606 2222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.856724 2222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:50.856777 2222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:50.893607 2222 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:39007
I20260927 13:42:50.893642 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:39007 every 8 connection(s)
I20260927 13:42:50.894708 2222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:42:50.896298 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2222
I20260927 13:42:50.896486 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:42:50.896910 1975 external_mini_cluster.cc:1428] Setting key bc59ded11a9aa0ca112b92a194fea253
Sep 27 13:42:50 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
I20260927 13:42:50.914181 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:50.897132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:44075 (local address 127.1.237.254:46529)
0927 13:42:50.897323 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:50.897329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:50.898494 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 13:42:50.900579 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:50.900585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:50.900660 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:50.900751 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:50.902077 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.902782 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.904307 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.904717 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.904851 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:50.908589 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:50.908594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:50.908597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:50.908625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:50.910985 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:50.911899 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:50.911904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:50.911906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:50.912035 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:50.912434 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:50.912438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:50.912441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:50.912771 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:50.912918 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:50.913603 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 13:42:50.913814 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:42:50.915194 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:50.897616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46529 (local address 127.1.237.194:44075)
0927 13:42:50.898358 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:50.898400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:50.900349 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:50.900955 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:50.900963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:50.901425 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:50.901923 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:50.901938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.902920 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.902926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:50.904165 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:50.904175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:50.904925 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:50.904932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:50.905160 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:50.905790 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:50.905810 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:50.908394 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:50.911130 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:50.911140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:50.911202 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:50.911768 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:50.912144 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:50.912147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:50.912149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:50.912306 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:50.912900 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:50.912907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:50.913211 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 13:42:50.913444 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 13:42:50.913900 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":325,"threads_started":1}
I20260927 13:42:50.916553 2350 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46529
I20260927 13:42:50.916834 2350 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:50.917447 2350 heartbeater.cc:511] Master 127.1.237.254:46529 requested a full tablet report, sending...
I20260927 13:42:50.918839 2027 ts_manager.cc:194] Registered new tserver with Master: 90061e2659824ea78da2ea47abf64abf (127.1.237.194:39007)
I20260927 13:42:50.919658 2027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:44075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:42:50.933960 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:51.243166 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:51.243446 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:51.243619 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:51.250733 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:51.250857 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:51.251035 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:42:51.260092 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:51.261162 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:51.262326 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:51.274915 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:51.274904 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:51.278792 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:51.280071 2355 server_base.cc:1034] running on GCE node
I20260927 13:42:51.280593 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:51.281463 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:51.282642 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790516571282585 us; error 52 us; skew 500 ppm
I20260927 13:42:51.282898 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:51.286091 2355 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:42:51.287426 2355 webserver.cc:460] Webserver started at http://127.1.237.195:38635/ using document root <none> and password file <none>
I20260927 13:42:51.288048 2355 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:51.288169 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:51.288470 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:51.291047 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "3d04cf6624244c6bb2a85a6c8dff2816"
format_stamp: "Formatted at 2026-09-27 13:42:51 on dist-test-slave-7g3l"
server_key: "cddcbe1abbfa6ae0348a1a6650169e3d"
server_key_iv: "52641e5c6754720d63b11cc4c561fded"
server_key_version: "encryptionkey@0"
I20260927 13:42:51.291834 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "3d04cf6624244c6bb2a85a6c8dff2816"
format_stamp: "Formatted at 2026-09-27 13:42:51 on dist-test-slave-7g3l"
server_key: "cddcbe1abbfa6ae0348a1a6650169e3d"
server_key_iv: "52641e5c6754720d63b11cc4c561fded"
server_key_version: "encryptionkey@0"
I20260927 13:42:51.296290 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 13:42:51.299399 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:51.300473 2355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:42:51.300634 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "3d04cf6624244c6bb2a85a6c8dff2816"
format_stamp: "Formatted at 2026-09-27 13:42:51 on dist-test-slave-7g3l"
server_key: "cddcbe1abbfa6ae0348a1a6650169e3d"
server_key_iv: "52641e5c6754720d63b11cc4c561fded"
server_key_version: "encryptionkey@0"
I20260927 13:42:51.300808 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:51.326900 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:51.330691 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:51.330929 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:51.331745 2355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:51.332947 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:51.333010 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:51.333149 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:51.333204 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:51.366102 2355 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34103
I20260927 13:42:51.366179 2482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34103 every 8 connection(s)
I20260927 13:42:51.367233 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:42:51.369855 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2355
I20260927 13:42:51.370061 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:42:51.370476 1975 external_mini_cluster.cc:1428] Setting key e7f6943091d040ca1ea0304c7a3cb417
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:51.385937 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.369464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:53707 (local address 127.1.237.254:46529)
0927 13:42:51.369668 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.369674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.371359 (+ 1685us) server_negotiation.cc:407] Connection header received
0927 13:42:51.373298 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.373303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.373382 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.373494 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.375082 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.375806 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.377237 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.377632 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.377761 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.380951 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.380958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.380963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.381002 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.383142 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.383736 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.383740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.383742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.383878 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.384248 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.384255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.384258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.384593 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.384745 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.385423 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.385619 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:42:51.386822 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.369942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46529 (local address 127.1.237.195:53707)
0927 13:42:51.371204 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:51.371252 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:51.373061 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:51.373707 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:51.373715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:51.374379 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:51.374920 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:51.374936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.375943 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.375953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:51.377103 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:51.377114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.377833 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.377842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:51.378089 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.378613 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:51.378636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:51.380764 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:51.383296 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:51.383301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:51.383311 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:51.383651 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:51.383958 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:51.383960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:51.383962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:51.384136 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:51.384718 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:51.384725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:51.385087 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 13:42:51.385356 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 13:42:51.385787 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":354,"threads_started":1}
I20260927 13:42:51.387974 2483 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46529
I20260927 13:42:51.388238 2483 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:51.388859 2483 heartbeater.cc:511] Master 127.1.237.254:46529 requested a full tablet report, sending...
I20260927 13:42:51.390046 2027 ts_manager.cc:194] Registered new tserver with Master: 3d04cf6624244c6bb2a85a6c8dff2816 (127.1.237.195:34103)
I20260927 13:42:51.390777 2027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:53707
I20260927 13:42:51.397521 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:51.440034 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.425226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.1.237.193:37051)
0927 13:42:51.425850 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.425856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.425917 (+ 61us) server_negotiation.cc:407] Connection header received
0927 13:42:51.426055 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.426061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.426269 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.426389 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.427875 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.428784 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.429964 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.430441 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.430529 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.434377 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.434387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.434398 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.434429 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.437374 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.437984 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.437993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.437996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.438124 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.438501 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.438508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.438511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.438813 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.439092 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.439373 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.439605 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":339,"threads_started":1}
W20260927 13:42:51.441279 2131 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60354
I20260927 13:42:51.446255 2216 heartbeater.cc:503] Master 127.1.237.254:46529 was elected leader, sending a full tablet report...
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:51.482940 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.469528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.1.237.193:37051)
0927 13:42:51.469759 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.469765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.469993 (+ 228us) server_negotiation.cc:407] Connection header received
0927 13:42:51.470199 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.470211 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.470296 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.470409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.471687 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.472367 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.473532 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.473947 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.474092 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.477778 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.477782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.477785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.477814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.480603 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.481154 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.481161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.481166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.481311 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.481639 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.481644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.481647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.481883 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.482108 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.482409 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.482636 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:42:51.484402 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2088
I20260927 13:42:51.495700 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.485268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.1.237.193:37051)
0927 13:42:51.485470 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.485477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.485736 (+ 259us) server_negotiation.cc:407] Connection header received
0927 13:42:51.485824 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.485828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.485879 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.485953 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.487040 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.487821 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.488994 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.489372 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.489458 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.491322 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.491326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.491329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.491353 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.493777 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.494261 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.494264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.494266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.494374 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.494629 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.494632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.494634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.494838 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.494991 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.495229 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.495357 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:42:51.496417 2211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:51.577322 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2088
I20260927 13:42:51.593374 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2222
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:42:51.607318 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.594178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.1.237.194:39007)
0927 13:42:51.594683 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.594689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.594712 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:42:51.594764 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.594770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.594961 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.595077 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.596101 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.597039 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.598154 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.598652 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.598744 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.602302 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.602311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.602323 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.602348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.604964 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.605622 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.605628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.605630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.605734 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.606044 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.606046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.606048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.606316 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.606518 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.606729 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.606892 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260927 13:42:51.608443 2345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:51.686110 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2222
I20260927 13:42:51.707757 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2355
Sep 27 13:42:51 dist-test-slave-7g3l krb5kdc[1982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:42:51.722481 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:51.708729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.1.237.195:34103)
0927 13:42:51.709345 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:51.709352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:51.709380 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:42:51.709441 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:51.709447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:51.709654 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:51.709784 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:51.710921 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.712019 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.713227 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:51.713760 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:51.713858 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:51.717754 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:51.717765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:51.717777 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:51.717807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:51.720275 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.720810 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.720816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.720819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.720929 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:51.721220 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:51.721224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:51.721226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:51.721541 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:51.721741 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:51.721967 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 13:42:51.722184 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":342,"threads_started":1}
I20260927 13:42:51.723693 2478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:51.802263 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2355
I20260927 13:42:51.818508 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 1995
I20260927 13:42:51.819633 2057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:51.914358 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 1995
2026-09-27T13:42:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4728 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:42:52 dist-test-slave-7g3l krb5kdc[2504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:42:52 dist-test-slave-7g3l krb5kdc[2504](info): set up 2 sockets
Sep 27 13:42:52 dist-test-slave-7g3l krb5kdc[2504](info): commencing operation
krb5kdc: starting...
W20260927 13:42:52.986918 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:42:53 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:42:53.150370 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42937
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:53.446161 2517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:53.446424 2517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:53.446501 2517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:53.453293 2517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:42:53.453388 2517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:53.453433 2517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:53.453486 2517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:42:53.453529 2517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:42:53.462950 2517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:53.464215 2517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:53.465381 2517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:53.476490 2522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:53.476536 2523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:53.476450 2525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:53.480470 2517 server_base.cc:1034] running on GCE node
I20260927 13:42:53.481114 2517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:53.482347 2517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:53.483527 2517 hybrid_clock.cc:648] HybridClock initialized: now 1790516573483495 us; error 24 us; skew 500 ppm
I20260927 13:42:53.483800 2517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:53 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:53.493374 2517 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:42:53.494792 2517 webserver.cc:460] Webserver started at http://127.1.237.254:46181/ using document root <none> and password file <none>
I20260927 13:42:53.495486 2517 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:53.495576 2517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:53.495822 2517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:53.498498 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "da49e4b761a34b81919ac7e49a9e8637"
format_stamp: "Formatted at 2026-09-27 13:42:53 on dist-test-slave-7g3l"
server_key: "c6b38bd674bbbeb319c3d8fb4f34ada9"
server_key_iv: "686e28546346d75824018d957210e342"
server_key_version: "encryptionkey@0"
I20260927 13:42:53.499210 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "da49e4b761a34b81919ac7e49a9e8637"
format_stamp: "Formatted at 2026-09-27 13:42:53 on dist-test-slave-7g3l"
server_key: "c6b38bd674bbbeb319c3d8fb4f34ada9"
server_key_iv: "686e28546346d75824018d957210e342"
server_key_version: "encryptionkey@0"
I20260927 13:42:53.503791 2517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:42:53.506745 2532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:53.507948 2517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:42:53.508076 2517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "da49e4b761a34b81919ac7e49a9e8637"
format_stamp: "Formatted at 2026-09-27 13:42:53 on dist-test-slave-7g3l"
server_key: "c6b38bd674bbbeb319c3d8fb4f34ada9"
server_key_iv: "686e28546346d75824018d957210e342"
server_key_version: "encryptionkey@0"
I20260927 13:42:53.508217 2517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:53.522773 2517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:53.526218 2517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:53.526424 2517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:53.544243 2517 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:42937
I20260927 13:42:53.544250 2583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:42937 every 8 connection(s)
I20260927 13:42:53.545424 2517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:42:53.548688 2584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:53.553339 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2517
I20260927 13:42:53.553578 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:42:53.554281 1975 external_mini_cluster.cc:1428] Setting key ec99a1fc5e91949933e9f2d1651e8783
I20260927 13:42:53.559223 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637: Bootstrap starting.
I20260927 13:42:53.562566 2584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:53.563728 2584 log.cc:824] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:53.567044 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637: No bootstrap required, opened a new log
Sep 27 13:42:53 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:53.575696 2584 raft_consensus.cc:348] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } }
I20260927 13:42:53.576145 2584 raft_consensus.cc:374] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:53.576270 2584 raft_consensus.cc:729] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da49e4b761a34b81919ac7e49a9e8637, State: Initialized, Role: FOLLOWER
I20260927 13:42:53.576925 2584 consensus_queue.cc:260] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } }
I20260927 13:42:53.577150 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:53.556137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.1.237.254:42937)
0927 13:42:53.556899 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:53.556909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:53.556949 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:42:53.558552 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:53.558574 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:53.558905 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:53.559285 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:53.560837 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:53.562030 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:53.563782 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:53.564748 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:53.564947 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:53.569544 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:53.569554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:53.569579 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:53.569621 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:53.572734 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:53.573442 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:53.573448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:53.573455 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:53.573634 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:53.574002 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:53.574006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:53.574008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:53.574704 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:53.575050 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:53.575543 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 13:42:53.575947 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":374,"threads_started":1}
I20260927 13:42:53.577167 2584 raft_consensus.cc:388] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:42:53.577402 2584 raft_consensus.cc:482] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:42:53.577605 2584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:53.580363 2584 raft_consensus.cc:504] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } }
I20260927 13:42:53.580773 2584 leader_election.cc:302] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da49e4b761a34b81919ac7e49a9e8637; no voters:
I20260927 13:42:53.581259 2584 leader_election.cc:288] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:42:53.581421 2589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:53.582052 2589 raft_consensus.cc:686] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [term 1 LEADER]: Becoming Leader. State: Replica: da49e4b761a34b81919ac7e49a9e8637, State: Running, Role: LEADER
I20260927 13:42:53.582576 2589 consensus_queue.cc:237] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } }
I20260927 13:42:53.583345 2584 sys_catalog.cc:564] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:42:53.585600 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da49e4b761a34b81919ac7e49a9e8637. Latest consensus state: current_term: 1 leader_uuid: "da49e4b761a34b81919ac7e49a9e8637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } } }
I20260927 13:42:53.585599 2590 sys_catalog.cc:455] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da49e4b761a34b81919ac7e49a9e8637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da49e4b761a34b81919ac7e49a9e8637" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42937 } } }
I20260927 13:42:53.586001 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [sys.catalog]: This master's current role is: LEADER
I20260927 13:42:53.586046 2590 sys_catalog.cc:458] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637 [sys.catalog]: This master's current role is: LEADER
I20260927 13:42:53.587430 2595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:42:53.593917 2595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:42:53.605247 2595 catalog_manager.cc:1347] Generated new cluster ID: f6739ed0ddd44904b5ca25261f0a8883
I20260927 13:42:53.605366 2595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:42:53.640043 2595 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:42:53.641413 2595 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:42:53.652374 2595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da49e4b761a34b81919ac7e49a9e8637: Generated new TSK 0
I20260927 13:42:53.653153 2595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:42:53.694785 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:53.994388 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:53.994640 2610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:53.994815 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:54.001787 2610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:54.001905 2610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:54.002094 2610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:42:54.011320 2610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:54.012666 2610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:54.013878 2610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:54.025936 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.025916 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.026161 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:54.026641 2610 server_base.cc:1034] running on GCE node
I20260927 13:42:54.027132 2610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:54.027961 2610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:54.029147 2610 hybrid_clock.cc:648] HybridClock initialized: now 1790516574029103 us; error 38 us; skew 500 ppm
I20260927 13:42:54.029500 2610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:54 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:54.032356 2610 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:42:54.033667 2610 webserver.cc:460] Webserver started at http://127.1.237.193:44503/ using document root <none> and password file <none>
I20260927 13:42:54.034274 2610 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:54.034392 2610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:54.034691 2610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:54.037362 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "c4b0b0d064b648e8aee312574248bf00"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "269d44aca78b6d6e5385a2be2e0badaa"
server_key_iv: "98be8f92c988e148a375d70de347eb4a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.038064 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "c4b0b0d064b648e8aee312574248bf00"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "269d44aca78b6d6e5385a2be2e0badaa"
server_key_iv: "98be8f92c988e148a375d70de347eb4a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.042483 2610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:42:54.045547 2625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.046765 2610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:42:54.046907 2610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "c4b0b0d064b648e8aee312574248bf00"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "269d44aca78b6d6e5385a2be2e0badaa"
server_key_iv: "98be8f92c988e148a375d70de347eb4a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.047076 2610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:54.065732 2610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:54.069321 2610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:54.069584 2610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:54.070396 2610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:54.071545 2610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:54.071619 2610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.071754 2610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:54.071811 2610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.102386 2610 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:40835
I20260927 13:42:54.102480 2737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:40835 every 8 connection(s)
I20260927 13:42:54.103626 2610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:42:54.109001 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2610
I20260927 13:42:54.109215 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:42:54.109774 1975 external_mini_cluster.cc:1428] Setting key 0cb76e868da1474479af889404218780
Sep 27 13:42:54 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:54.123477 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:54.105889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:42917 (local address 127.1.237.254:42937)
0927 13:42:54.106312 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:54.106316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:54.107172 (+ 856us) server_negotiation.cc:407] Connection header received
0927 13:42:54.108836 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:54.108842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:54.108925 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:54.109038 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:54.110459 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.111464 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.112948 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.113334 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.113466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:54.116555 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:54.116562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:54.116567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:54.116606 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:54.120089 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:54.120972 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:54.120980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:54.120986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:54.121134 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:54.121610 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:54.121616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:54.121622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:54.121948 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:54.122095 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:54.122741 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 13:42:54.122960 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":228,"threads_started":1}
I20260927 13:42:54.124042 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:54.106359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42937 (local address 127.1.237.193:42917)
0927 13:42:54.106992 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:54.107032 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:54.108591 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:54.109260 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:54.109268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:54.109755 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:54.110303 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:54.110319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.111603 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.111613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:54.112811 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:54.112823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.113542 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.113550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:54.113752 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:54.114402 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:54.114426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:54.116395 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:54.120234 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:54.120244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:54.120260 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:54.120838 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:54.121260 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:54.121267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:54.121274 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:54.121490 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:54.122071 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:54.122078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:54.122409 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 13:42:54.122623 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 13:42:54.122921 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":249,"threads_started":1}
I20260927 13:42:54.125499 2738 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42937
I20260927 13:42:54.125785 2738 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:42:54.126524 2738 heartbeater.cc:511] Master 127.1.237.254:42937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:42:54.128537 2549 ts_manager.cc:194] Registered new tserver with Master: c4b0b0d064b648e8aee312574248bf00 (127.1.237.193:40835)
I20260927 13:42:54.130716 2549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:42917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:42:54.147440 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:54.436383 2744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:54.436640 2744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:54.436794 2744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:54.443090 2744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:54.443274 2744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:54.443444 2744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:42:54.452385 2744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:54.453492 2744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:54.454706 2744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:54.466972 2749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.466933 2750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.467015 2752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:54.467679 2744 server_base.cc:1034] running on GCE node
I20260927 13:42:54.468137 2744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:54.468932 2744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:54.470134 2744 hybrid_clock.cc:648] HybridClock initialized: now 1790516574470092 us; error 38 us; skew 500 ppm
I20260927 13:42:54.470366 2744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:54 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:54.473278 2744 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:42:54.474561 2744 webserver.cc:460] Webserver started at http://127.1.237.194:43001/ using document root <none> and password file <none>
I20260927 13:42:54.475195 2744 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:54.475338 2744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:54.475620 2744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:54.478168 2744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "0283c780f5844b6f876e03b10862b0e3"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "2d16d77363e2bb6236521a4927fe8260"
server_key_iv: "f1a9504001caca94ba94c858d3561e9a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.478874 2744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "0283c780f5844b6f876e03b10862b0e3"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "2d16d77363e2bb6236521a4927fe8260"
server_key_iv: "f1a9504001caca94ba94c858d3561e9a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.483345 2744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:42:54.486339 2759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.487530 2744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:42:54.487685 2744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "0283c780f5844b6f876e03b10862b0e3"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "2d16d77363e2bb6236521a4927fe8260"
server_key_iv: "f1a9504001caca94ba94c858d3561e9a"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.487859 2744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:54.501610 2744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:54.505487 2744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:54.505750 2744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:54.506567 2744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:54.507807 2744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:54.507902 2744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.508037 2744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:54.508118 2744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.538676 2744 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:39271
I20260927 13:42:54.538746 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:39271 every 8 connection(s)
I20260927 13:42:54.540197 2744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:42:54.550071 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2744
I20260927 13:42:54.550282 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:42:54.550833 1975 external_mini_cluster.cc:1428] Setting key 073cfd5949c891481c7830630dd4a84a
Sep 27 13:42:54 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:54.560243 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:54.542370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:44891 (local address 127.1.237.254:42937)
0927 13:42:54.542553 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:54.542567 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:54.543611 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 13:42:54.545486 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:54.545491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:54.545594 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:54.545704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:54.546984 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.547762 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.549250 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.549699 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.549828 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:54.553866 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:54.553873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:54.553878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:54.553914 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:54.557232 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:54.558088 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:54.558094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:54.558097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:54.558242 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:54.558602 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:54.558608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:54.558613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:54.558950 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:54.559088 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:54.559731 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 13:42:54.559930 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:42:54.561120 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:54.542763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42937 (local address 127.1.237.194:44891)
0927 13:42:54.543465 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:54.543504 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:54.545232 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:54.545912 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:54.545919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:54.546339 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:54.546824 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:54.546839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.547912 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.547916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:54.549106 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:54.549116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:54.549906 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:54.549913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:54.550265 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:54.550914 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:54.550940 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:54.553660 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:54.557379 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:54.557388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:54.557404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:54.557972 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:54.558350 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:54.558353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:54.558355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:54.558507 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:54.559055 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:54.559060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:54.559377 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 13:42:54.559614 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 13:42:54.560040 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":240,"threads_started":1}
I20260927 13:42:54.562323 2872 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42937
I20260927 13:42:54.562593 2872 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:54.563401 2872 heartbeater.cc:511] Master 127.1.237.254:42937 requested a full tablet report, sending...
I20260927 13:42:54.564785 2549 ts_manager.cc:194] Registered new tserver with Master: 0283c780f5844b6f876e03b10862b0e3 (127.1.237.194:39271)
I20260927 13:42:54.565958 2549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:44891
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:42:54.589445 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:54.881115 2877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:54.881340 2877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:54.881424 2877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:54.887806 2877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:54.887884 2877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:54.887984 2877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:42:54.896471 2877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:54.897377 2877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:54.898411 2877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:54.910351 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.910262 2885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:54.910355 2882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:54.910890 2877 server_base.cc:1034] running on GCE node
I20260927 13:42:54.911558 2877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:54.912479 2877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:54.913687 2877 hybrid_clock.cc:648] HybridClock initialized: now 1790516574913654 us; error 52 us; skew 500 ppm
I20260927 13:42:54.913972 2877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:54 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:54.916842 2877 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:42:54.918037 2877 webserver.cc:460] Webserver started at http://127.1.237.195:34385/ using document root <none> and password file <none>
I20260927 13:42:54.918655 2877 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:54.918787 2877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:54.919063 2877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:54.921639 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "12511dd3e19d49e285903c1303f5dd0d"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "1da19304f5799c6c9628549bcebc980e"
server_key_iv: "2cb69af0dd8b2e7ec121f960bcaed83c"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.922369 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "12511dd3e19d49e285903c1303f5dd0d"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "1da19304f5799c6c9628549bcebc980e"
server_key_iv: "2cb69af0dd8b2e7ec121f960bcaed83c"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.926867 2877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:42:54.929831 2892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.930894 2877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:42:54.931044 2877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "12511dd3e19d49e285903c1303f5dd0d"
format_stamp: "Formatted at 2026-09-27 13:42:54 on dist-test-slave-7g3l"
server_key: "1da19304f5799c6c9628549bcebc980e"
server_key_iv: "2cb69af0dd8b2e7ec121f960bcaed83c"
server_key_version: "encryptionkey@0"
I20260927 13:42:54.931273 2877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:54.965036 2877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:54.968921 2877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:54.969182 2877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:54.970033 2877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:42:54.971310 2877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:54.971410 2877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:54.971546 2877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:54.971618 2877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:55.002545 2877 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34183
I20260927 13:42:55.002671 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34183 every 8 connection(s)
I20260927 13:42:55.003842 2877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:42:55.013060 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 2877
I20260927 13:42:55.013243 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:42:55.013762 1975 external_mini_cluster.cc:1428] Setting key 378bb92edf53b646bc027eb1e496b224
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:55.022449 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.006210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:56173 (local address 127.1.237.254:42937)
0927 13:42:55.006401 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.006408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.007524 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 13:42:55.009214 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.009218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.009281 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.009380 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.010674 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.011488 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.013047 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.013702 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.013827 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.017324 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.017328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.017331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.017357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.020056 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.020622 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.020626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.020628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.020727 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.021037 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.021040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.021042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.021271 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.021371 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.021930 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.022131 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:42:55.023797 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.006694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42937 (local address 127.1.237.195:56173)
0927 13:42:55.007378 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.007421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.008990 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.009585 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.009592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.010033 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:55.010529 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.010547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.011655 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.011660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.012912 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.012923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.013917 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.013922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.014099 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.014748 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:55.014772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:55.017169 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:55.020167 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:55.020173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:55.020183 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:55.020534 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:55.020810 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:55.020813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:55.020815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:55.020957 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:55.021367 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:55.021371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:55.021618 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.021852 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.022298 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":252,"threads_started":1}
I20260927 13:42:55.024966 3005 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42937
I20260927 13:42:55.025192 3005 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:55.025771 3005 heartbeater.cc:511] Master 127.1.237.254:42937 requested a full tablet report, sending...
I20260927 13:42:55.026968 2549 ts_manager.cc:194] Registered new tserver with Master: 12511dd3e19d49e285903c1303f5dd0d (127.1.237.195:34183)
I20260927 13:42:55.027853 2549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:56173
I20260927 13:42:55.028744 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:42:55.046248 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.034398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.1.237.254:42937)
0927 13:42:55.034550 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.034564 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.034947 (+ 383us) server_negotiation.cc:407] Connection header received
0927 13:42:55.035293 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.035300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.035384 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.035486 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.036896 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.037587 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.038795 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.039214 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.039342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.041372 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.041378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.041384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.041416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.043893 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.044448 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.044453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.044456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.044569 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.044916 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.044922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.044925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.045201 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.045437 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.045765 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.045955 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 13:42:55.051728 2549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:42:55.058341 2549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:42:55.080140 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.070140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.1.237.195:34183)
0927 13:42:55.072711 (+ 2571us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.072718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.072994 (+ 276us) server_negotiation.cc:407] Connection header received
0927 13:42:55.073158 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.073166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.073372 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.073543 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:55.074312 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.075520 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.077699 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.079032 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.079191 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.079405 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.079667 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.079826 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2410,"thread_start_us":2274,"threads_started":1}
I20260927 13:42:55.080173 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.069642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34183 (local address 127.0.0.1:58238)
0927 13:42:55.072895 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.072918 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.073024 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.073569 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.073576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.073604 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:55.074124 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.074140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.075648 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.075653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.077557 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.077574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.079257 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.079262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.079448 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.079504 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.079620 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.079786 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3135,"mutex_wait_us":6,"thread_start_us":2149,"threads_started":1}
I20260927 13:42:55.081360 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.069388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:39271 (local address 127.0.0.1:46822)
0927 13:42:55.072229 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.072275 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.072519 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.073686 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.073690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.073715 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:55.075241 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.075253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.076518 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.076526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.078725 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.078738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.080631 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.080637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.080785 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.080806 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.080904 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.081015 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2564,"mutex_wait_us":656,"thread_start_us":989,"threads_started":1}
I20260927 13:42:55.081352 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.069476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.1.237.194:39271)
0927 13:42:55.071847 (+ 2371us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.071855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.072377 (+ 522us) server_negotiation.cc:407] Connection header received
0927 13:42:55.072750 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.072758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.073002 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.073163 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:55.075383 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.076394 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.078883 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.080206 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.080348 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.080498 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.080942 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.081095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2218,"thread_start_us":2139,"threads_started":1}
I20260927 13:42:55.082244 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.070222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:40835 (local address 127.0.0.1:60714)
0927 13:42:55.073016 (+ 2794us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.073037 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.073202 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.073963 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.073968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.073990 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:55.074418 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.074429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.076985 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.076992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.079255 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.079266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.081470 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.081485 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.081661 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.081689 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.081775 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.081880 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2663,"mutex_wait_us":1770,"thread_start_us":721,"threads_started":1}
I20260927 13:42:55.083086 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.070671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.1.237.193:40835)
0927 13:42:55.072243 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.072252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.073114 (+ 862us) server_negotiation.cc:407] Connection header received
0927 13:42:55.073376 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.073382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.073570 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.073744 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:55.075417 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.076855 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.079423 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.081009 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.081107 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.081274 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.082657 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.082800 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1426,"thread_start_us":1307,"threads_started":1}
I20260927 13:42:55.084333 2940 tablet_service.cc:1467] Processing CreateTablet for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 (DEFAULT_TABLE table=test-table [id=2b1f257221a342dc86d8371e64fa4a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:55.084542 2807 tablet_service.cc:1467] Processing CreateTablet for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 (DEFAULT_TABLE table=test-table [id=2b1f257221a342dc86d8371e64fa4a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:55.085597 2807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:55.085726 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:55.086270 2673 tablet_service.cc:1467] Processing CreateTablet for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 (DEFAULT_TABLE table=test-table [id=2b1f257221a342dc86d8371e64fa4a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:55.087894 2673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:55.096297 3023 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Bootstrap starting.
I20260927 13:42:55.099069 3023 tablet_bootstrap.cc:654] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:55.100384 3023 log.cc:824] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:55.102764 3026 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Bootstrap starting.
I20260927 13:42:55.103627 3023 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: No bootstrap required, opened a new log
I20260927 13:42:55.103845 3023 ts_tablet_manager.cc:1397] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 13:42:55.107101 3026 tablet_bootstrap.cc:654] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:55.108559 3026 log.cc:824] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:55.110085 3024 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Bootstrap starting.
I20260927 13:42:55.112879 3024 tablet_bootstrap.cc:654] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:55.113294 3023 raft_consensus.cc:348] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.113670 3023 raft_consensus.cc:374] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:55.113801 3024 log.cc:824] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:55.113808 3023 raft_consensus.cc:729] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0283c780f5844b6f876e03b10862b0e3, State: Initialized, Role: FOLLOWER
I20260927 13:42:55.114389 3023 consensus_queue.cc:260] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.118640 2872 heartbeater.cc:503] Master 127.1.237.254:42937 was elected leader, sending a full tablet report...
I20260927 13:42:55.119292 3023 ts_tablet_manager.cc:1428] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260927 13:42:55.119647 3026 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: No bootstrap required, opened a new log
I20260927 13:42:55.120182 3024 tablet_bootstrap.cc:492] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: No bootstrap required, opened a new log
I20260927 13:42:55.120316 3026 ts_tablet_manager.cc:1397] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260927 13:42:55.120432 3024 ts_tablet_manager.cc:1397] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260927 13:42:55.127628 3024 raft_consensus.cc:348] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.127909 3024 raft_consensus.cc:374] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:55.127722 3026 raft_consensus.cc:348] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.128013 3024 raft_consensus.cc:729] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12511dd3e19d49e285903c1303f5dd0d, State: Initialized, Role: FOLLOWER
I20260927 13:42:55.128062 3026 raft_consensus.cc:374] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:55.128168 3026 raft_consensus.cc:729] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b0b0d064b648e8aee312574248bf00, State: Initialized, Role: FOLLOWER
I20260927 13:42:55.128561 3024 consensus_queue.cc:260] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.129143 3026 consensus_queue.cc:260] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.129570 3005 heartbeater.cc:503] Master 127.1.237.254:42937 was elected leader, sending a full tablet report...
I20260927 13:42:55.130072 3024 ts_tablet_manager.cc:1428] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 13:42:55.130133 2738 heartbeater.cc:503] Master 127.1.237.254:42937 was elected leader, sending a full tablet report...
I20260927 13:42:55.130673 3026 ts_tablet_manager.cc:1428] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260927 13:42:55.181993 3030 raft_consensus.cc:482] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:42:55.182252 3030 raft_consensus.cc:504] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.183795 3030 leader_election.cc:288] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0283c780f5844b6f876e03b10862b0e3 (127.1.237.194:39271), c4b0b0d064b648e8aee312574248bf00 (127.1.237.193:40835)
I20260927 13:42:55.189008 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.184396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:33665 (local address 127.1.237.193:40835)
0927 13:42:55.184594 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.184598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.184612 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:42:55.184692 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.184696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.184754 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.184844 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:55.185192 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.185849 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.187424 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.188186 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.188341 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.188409 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.188662 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.188772 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:42:55.189008 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.184024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:40835 (local address 127.1.237.195:33665)
0927 13:42:55.184272 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.184292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.184411 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.184864 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.184869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.184893 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:55.185058 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.185067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.185974 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.185981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.187268 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.187279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.188335 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.188341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.188511 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.188533 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.188632 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.188753 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":20}
I20260927 13:42:55.190019 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e9f6ba99bd4e60b1b4a8f8b6c2f3f1" candidate_uuid: "12511dd3e19d49e285903c1303f5dd0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b0b0d064b648e8aee312574248bf00" is_pre_election: true
I20260927 13:42:55.190364 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.184085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:43965 (local address 127.1.237.194:39271)
0927 13:42:55.184281 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.184288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.184947 (+ 659us) server_negotiation.cc:407] Connection header received
0927 13:42:55.185144 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.185153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.185206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.185295 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:55.185746 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.186399 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.188515 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.189444 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.189597 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.189643 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.189991 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.190118 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:42:55.190440 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.184153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:39271 (local address 127.1.237.195:43965)
0927 13:42:55.184838 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:55.184859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:55.184992 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:55.185308 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:55.185312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:55.185332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:55.185608 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.185616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.186560 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.186566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.188370 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:55.188380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.189655 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.189663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:55.189828 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.189858 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:42:55.189956 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:42:55.190098 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1,"thread_start_us":422,"threads_started":1}
I20260927 13:42:55.190518 2693 raft_consensus.cc:2454] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12511dd3e19d49e285903c1303f5dd0d in term 0.
I20260927 13:42:55.191210 2893 leader_election.cc:302] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12511dd3e19d49e285903c1303f5dd0d, c4b0b0d064b648e8aee312574248bf00; no voters:
I20260927 13:42:55.191303 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e9f6ba99bd4e60b1b4a8f8b6c2f3f1" candidate_uuid: "12511dd3e19d49e285903c1303f5dd0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0283c780f5844b6f876e03b10862b0e3" is_pre_election: true
I20260927 13:42:55.191567 3030 raft_consensus.cc:2781] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:42:55.191699 3030 raft_consensus.cc:482] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:42:55.191751 2827 raft_consensus.cc:2454] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12511dd3e19d49e285903c1303f5dd0d in term 0.
I20260927 13:42:55.191818 3030 raft_consensus.cc:3037] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:55.193922 3030 raft_consensus.cc:504] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.194393 3030 leader_election.cc:288] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [CANDIDATE]: Term 1 election: Requested vote from peers 0283c780f5844b6f876e03b10862b0e3 (127.1.237.194:39271), c4b0b0d064b648e8aee312574248bf00 (127.1.237.193:40835)
I20260927 13:42:55.194702 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e9f6ba99bd4e60b1b4a8f8b6c2f3f1" candidate_uuid: "12511dd3e19d49e285903c1303f5dd0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0283c780f5844b6f876e03b10862b0e3"
I20260927 13:42:55.194746 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e9f6ba99bd4e60b1b4a8f8b6c2f3f1" candidate_uuid: "12511dd3e19d49e285903c1303f5dd0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b0b0d064b648e8aee312574248bf00"
I20260927 13:42:55.194890 2827 raft_consensus.cc:3037] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:55.194902 2693 raft_consensus.cc:3037] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:55.197791 2693 raft_consensus.cc:2454] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12511dd3e19d49e285903c1303f5dd0d in term 1.
I20260927 13:42:55.197791 2827 raft_consensus.cc:2454] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12511dd3e19d49e285903c1303f5dd0d in term 1.
I20260927 13:42:55.198166 2893 leader_election.cc:302] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12511dd3e19d49e285903c1303f5dd0d, c4b0b0d064b648e8aee312574248bf00; no voters:
I20260927 13:42:55.198448 3030 raft_consensus.cc:2781] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:55.198850 3030 raft_consensus.cc:686] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [term 1 LEADER]: Becoming Leader. State: Replica: 12511dd3e19d49e285903c1303f5dd0d, State: Running, Role: LEADER
I20260927 13:42:55.199364 3030 consensus_queue.cc:237] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } }
I20260927 13:42:55.203521 2547 catalog_manager.cc:5495] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 12511dd3e19d49e285903c1303f5dd0d (127.1.237.195). New cstate: current_term: 1 leader_uuid: "12511dd3e19d49e285903c1303f5dd0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0283c780f5844b6f876e03b10862b0e3" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12511dd3e19d49e285903c1303f5dd0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4b0b0d064b648e8aee312574248bf00" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 40835 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:55.235831 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.222155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.1.237.193:40835)
0927 13:42:55.222326 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.222333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.222777 (+ 444us) server_negotiation.cc:407] Connection header received
0927 13:42:55.222972 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.222976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.223032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.223126 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.224480 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.225174 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.226373 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.226770 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.226924 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.230894 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.230905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.230917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.230947 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.233453 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.234021 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.234027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.234029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.234187 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.234472 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.234475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.234477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.234783 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.235039 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.235374 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.235506 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:42:55.255416 3006 tablet.cc:2369] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 12511dd3e19d49e285903c1303f5dd0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:55.278085 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.265581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.1.237.193:40835)
0927 13:42:55.265795 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.265803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.265919 (+ 116us) server_negotiation.cc:407] Connection header received
0927 13:42:55.266096 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.266100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.266161 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.266249 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.267445 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.268124 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.269252 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.269630 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.269714 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.272961 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.272965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.272968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.272994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.275613 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.276078 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.276082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.276084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.276202 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.276930 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.276934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.276935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.277196 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.277376 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.277582 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.277724 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 13:42:55.291998 2873 tablet.cc:2369] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P 0283c780f5844b6f876e03b10862b0e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:55.316797 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.304248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.1.237.193:40835)
0927 13:42:55.304442 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.304449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.304754 (+ 305us) server_negotiation.cc:407] Connection header received
0927 13:42:55.304933 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.304936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.304989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.305101 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.306304 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.306941 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.308109 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.308535 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.308660 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.312520 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.312524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.312527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.312554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.314891 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.315463 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.315466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.315468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.315570 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.315838 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.315841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.315842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.316084 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.316249 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.316427 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.316563 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 13:42:55.318555 2653 tablet_service.cc:2908] Rejecting scan request for tablet 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1: Uninitialized: clean time has not yet been initialized
I20260927 13:42:55.321412 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2610
I20260927 13:42:55.331821 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.322099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.1.237.193:40835)
0927 13:42:55.322282 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.322287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.322454 (+ 167us) server_negotiation.cc:407] Connection header received
0927 13:42:55.322539 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.322542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.322600 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.322668 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.323676 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.324270 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.325303 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.325642 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.325731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.327517 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.327521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.327523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.327547 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.329752 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.330339 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.330342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.330344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.330432 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.330707 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.330709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.330711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.330924 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.331109 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.331354 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.331482 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:42:55.332369 2733 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 13:42:55.415467 2739 tablet.cc:2369] T 69e9f6ba99bd4e60b1b4a8f8b6c2f3f1 P c4b0b0d064b648e8aee312574248bf00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:42:55.416116 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2610
I20260927 13:42:55.431736 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2744
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:42:55.448726 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.432619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.1.237.194:39271)
0927 13:42:55.432810 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.432816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.432834 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:42:55.432894 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.432898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.432965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.433086 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.434223 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.435032 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.437160 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.437874 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.438004 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.443156 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.443192 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.443211 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.443273 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.446478 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.447030 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.447036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.447038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.447217 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.447531 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.447534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.447536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.447846 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.448058 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.448237 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.448374 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:42:55.449352 2867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:55.537686 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2744
I20260927 13:42:55.553925 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2877
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:42:55.567786 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:55.554768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.1.237.195:34183)
0927 13:42:55.554946 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:55.554954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:55.554976 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:42:55.555042 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:55.555046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:55.555117 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:55.555272 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:55.556430 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.557122 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.558517 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:55.558926 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:55.559052 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:55.562839 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:55.562852 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:55.562866 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:55.562895 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:55.565299 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.565883 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.565890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.565892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.566024 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:55.566379 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:55.566386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:55.566389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:55.566697 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:55.566960 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:55.567343 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 13:42:55.567554 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:42:55.568384 3000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:55.654369 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2877
I20260927 13:42:55.669656 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 2517
I20260927 13:42:55.670678 2579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:42:55.770869 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 2517
2026-09-27T13:42:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3853 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[3052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[3052](info): set up 2 sockets
Sep 27 13:42:55 dist-test-slave-7g3l krb5kdc[3052](info): commencing operation
krb5kdc: starting...
W20260927 13:42:56.806865 1975 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:42:56 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:42:56.966203 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:35815
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:57.261109 3065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:57.261410 3065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:57.261581 3065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:57.268393 3065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:42:57.268525 3065 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:42:57.268605 3065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:57.268708 3065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:57.268783 3065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:42:57.268865 3065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:42:57.277791 3065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:35815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:57.278899 3065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:57.280118 3065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:57.291030 3070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:57.291124 3073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:57.291102 3071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:57.291903 3065 server_base.cc:1034] running on GCE node
I20260927 13:42:57.292606 3065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:57.293779 3065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:57.295001 3065 hybrid_clock.cc:648] HybridClock initialized: now 1790516577294938 us; error 27 us; skew 500 ppm
I20260927 13:42:57.295326 3065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:57 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:57.298213 3065 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:42:57.299484 3065 webserver.cc:460] Webserver started at http://127.1.237.254:38887/ using document root <none> and password file <none>
I20260927 13:42:57.300098 3065 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:57.300210 3065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:57.300590 3065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:57.303279 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "66c6638cc03a49758b299071f57acd79"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "416846dbe79a8239e0a4078b1faf510d"
server_key_iv: "986aef44ab491dfe44db4e070ef8abc7"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.304001 3065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "66c6638cc03a49758b299071f57acd79"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "416846dbe79a8239e0a4078b1faf510d"
server_key_iv: "986aef44ab491dfe44db4e070ef8abc7"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.308650 3065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:42:57.311887 3080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:57.313145 3065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:42:57.313308 3065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "66c6638cc03a49758b299071f57acd79"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "416846dbe79a8239e0a4078b1faf510d"
server_key_iv: "986aef44ab491dfe44db4e070ef8abc7"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.313503 3065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:57.328066 3065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:57.331800 3065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:57.332090 3065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:57.352918 3065 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:35815
I20260927 13:42:57.352923 3141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:35815 every 8 connection(s)
I20260927 13:42:57.354271 3065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:42:57.357797 3142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:57.358999 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3065
I20260927 13:42:57.359256 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:42:57.360077 1975 external_mini_cluster.cc:1428] Setting key 6b426cf1cdb0a813ca8e2da135857b27
I20260927 13:42:57.369951 3142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Bootstrap starting.
Sep 27 13:42:57 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:57.373026 3142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:57.374131 3142 log.cc:824] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:57.377157 3142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: No bootstrap required, opened a new log
I20260927 13:42:57.380890 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:57.362000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.1.237.254:35815)
0927 13:42:57.362733 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:57.362744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:57.362784 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:42:57.364360 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:57.364380 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:57.364755 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:57.365117 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:57.366443 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.367765 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.368935 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.369555 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.369739 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:57.373838 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:57.373844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:57.373856 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:57.373898 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:57.377291 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.378012 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.378018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.378025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.378164 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.378533 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.378536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.378538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.378995 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:57.379261 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:57.379657 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 13:42:57.379982 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260927 13:42:57.384807 3142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } }
I20260927 13:42:57.385098 3142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:57.385197 3142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c6638cc03a49758b299071f57acd79, State: Initialized, Role: FOLLOWER
I20260927 13:42:57.385679 3142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } }
I20260927 13:42:57.385900 3142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:42:57.386040 3142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:42:57.386193 3142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:57.388154 3142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } }
I20260927 13:42:57.388576 3142 leader_election.cc:302] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c6638cc03a49758b299071f57acd79; no voters:
I20260927 13:42:57.389140 3142 leader_election.cc:288] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:42:57.389242 3147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:57.389909 3147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [term 1 LEADER]: Becoming Leader. State: Replica: 66c6638cc03a49758b299071f57acd79, State: Running, Role: LEADER
I20260927 13:42:57.390389 3147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } }
I20260927 13:42:57.391059 3142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:42:57.393282 3149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c6638cc03a49758b299071f57acd79. Latest consensus state: current_term: 1 leader_uuid: "66c6638cc03a49758b299071f57acd79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } } }
I20260927 13:42:57.393257 3148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c6638cc03a49758b299071f57acd79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c6638cc03a49758b299071f57acd79" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 35815 } } }
I20260927 13:42:57.393757 3148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [sys.catalog]: This master's current role is: LEADER
I20260927 13:42:57.393980 3149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79 [sys.catalog]: This master's current role is: LEADER
I20260927 13:42:57.395836 3153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:42:57.400470 3153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:42:57.408349 3153 catalog_manager.cc:1347] Generated new cluster ID: d26890a15ea04d879bcbf192e9d0a13e
I20260927 13:42:57.408448 3153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:42:57.418864 3153 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:42:57.420006 3153 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:42:57.426414 3153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Generated new TSK 0
I20260927 13:42:57.427042 3153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:42:57.480183 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:35815
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:57.800953 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:57.801210 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:57.801301 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:57.808243 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:57.808339 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:57.808419 3168 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:42:57.808502 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:42:57.817960 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:57.819020 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:57.820286 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:57.832459 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:57.832428 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:57.832942 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:57.833360 3168 server_base.cc:1034] running on GCE node
I20260927 13:42:57.833983 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:57.834880 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:57.836138 3168 hybrid_clock.cc:648] HybridClock initialized: now 1790516577836051 us; error 63 us; skew 500 ppm
I20260927 13:42:57.836457 3168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:57 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:57.839538 3168 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:42:57.840940 3168 webserver.cc:460] Webserver started at http://127.1.237.193:36167/ using document root <none> and password file <none>
I20260927 13:42:57.841565 3168 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:57.841679 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:57.841987 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:57.844800 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "1296e5d8884f46f48ad9f669892684b8"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "670dbf585e2e545993ac7abc2d1db405"
server_key_iv: "536fcded78e4d1ba77dbb08c7563c787"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.845551 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "1296e5d8884f46f48ad9f669892684b8"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "670dbf585e2e545993ac7abc2d1db405"
server_key_iv: "536fcded78e4d1ba77dbb08c7563c787"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.850196 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:42:57.853400 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:57.854615 3168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:42:57.854764 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "1296e5d8884f46f48ad9f669892684b8"
format_stamp: "Formatted at 2026-09-27 13:42:57 on dist-test-slave-7g3l"
server_key: "670dbf585e2e545993ac7abc2d1db405"
server_key_iv: "536fcded78e4d1ba77dbb08c7563c787"
server_key_version: "encryptionkey@0"
I20260927 13:42:57.854956 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:57.875272 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:57.879041 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:57.879319 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:57.881691 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:57.881786 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:57.881953 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:57.882351 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:42:57 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:57.903131 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:57.882113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:39909 (local address 127.1.237.254:35815)
0927 13:42:57.882538 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:57.882544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:57.883689 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 13:42:57.885643 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:57.885652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:57.885739 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:57.885865 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:57.888085 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.889312 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.891607 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.892081 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.892226 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:57.897439 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:57.897443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:57.897446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:57.897506 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:57.900130 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.901078 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.901082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.901084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.901202 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.901594 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.901603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.901605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.901838 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:57.901983 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:57.902590 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 13:42:57.902789 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":235,"threads_started":1}
I20260927 13:42:57.904330 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:57.882666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.193:39909)
0927 13:42:57.883527 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:57.883577 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:57.885280 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:57.886162 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:57.886175 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:57.886970 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:57.887824 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:57.887847 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.889466 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.889477 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:57.891447 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:57.891461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.892332 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.892342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:57.892683 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:57.893440 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:57.893487 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:57.897216 (+ 3729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:57.900309 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:57.900318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:57.900338 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:57.900917 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:57.901305 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:57.901311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:57.901317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:57.901486 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:57.901959 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:57.901965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:57.902256 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 13:42:57.902479 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 13:42:57.902933 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":388,"threads_started":1}
I20260927 13:42:57.932541 3168 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:44837
I20260927 13:42:57.932790 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:57.917443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:53463 (local address 127.1.237.254:35815)
0927 13:42:57.917656 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:57.917664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:57.918431 (+ 767us) server_negotiation.cc:407] Connection header received
0927 13:42:57.918627 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:57.918632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:57.918709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:57.918804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:57.920591 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.921741 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.924026 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.924762 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.924862 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:57.927448 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:57.927456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:57.927462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:57.927498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:57.929975 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.930819 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.930823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.930825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.930924 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:57.931353 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:57.931358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:57.931360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:57.931682 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:57.931833 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:57.932263 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 13:42:57.932475 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:42:57.933001 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:57.917533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.193:53463)
0927 13:42:57.918291 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:57.918340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:57.918499 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:57.918823 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:57.918827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:57.919845 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:57.920392 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:57.920407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.921916 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.921922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:57.923834 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:57.923857 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:57.925169 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:57.925178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:57.925408 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:57.926051 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:57.926079 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:57.927263 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:57.930100 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:57.930120 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:57.930126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:57.930709 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:57.931020 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:57.931024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:57.931027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:57.931212 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:57.931803 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:57.931808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:57.931991 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 13:42:57.932160 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 13:42:57.932499 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":368,"threads_started":1}
I20260927 13:42:57.932571 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:44837 every 8 connection(s)
I20260927 13:42:57.933892 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:42:57.935935 3304 heartbeater.cc:347] Connected to a master server at 127.1.237.254:35815
I20260927 13:42:57.935951 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3168
I20260927 13:42:57.936167 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:42:57.936308 3304 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:57.936868 1975 external_mini_cluster.cc:1428] Setting key 4d27957274047e73b986509607379e2f
I20260927 13:42:57.936873 3304 heartbeater.cc:511] Master 127.1.237.254:35815 requested a full tablet report, sending...
I20260927 13:42:57.938820 3096 ts_manager.cc:194] Registered new tserver with Master: 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193:44837)
I20260927 13:42:57.940176 3096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:39909
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:42:57.972188 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:35815
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:58.273564 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:58.273787 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:58.273867 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:58.280769 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:58.280874 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:58.280952 3308 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:42:58.281044 3308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:42:58.290228 3308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:58.291417 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:58.292788 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:58.305089 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:58.305136 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:58.305158 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:58.305836 3308 server_base.cc:1034] running on GCE node
I20260927 13:42:58.306401 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:58.307380 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:58.308580 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790516578308520 us; error 40 us; skew 500 ppm
I20260927 13:42:58.308851 3308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:58.311774 3308 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:42:58.313025 3308 webserver.cc:460] Webserver started at http://127.1.237.194:44787/ using document root <none> and password file <none>
I20260927 13:42:58.313658 3308 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:58.313772 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:58.314079 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:58.316809 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "c4228715b0894a5687f3122fa42872ec"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "1c2c6e3d083b19541a36a4ec7a1c808a"
server_key_iv: "4eae8c13a054949ba64a3c6ab07ccb0c"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.317543 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "c4228715b0894a5687f3122fa42872ec"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "1c2c6e3d083b19541a36a4ec7a1c808a"
server_key_iv: "4eae8c13a054949ba64a3c6ab07ccb0c"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.322144 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:42:58.325222 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:58.326329 3308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:42:58.326481 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "c4228715b0894a5687f3122fa42872ec"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "1c2c6e3d083b19541a36a4ec7a1c808a"
server_key_iv: "4eae8c13a054949ba64a3c6ab07ccb0c"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.326686 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:58.350011 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:58.353916 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:58.354146 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:58.356694 3308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:58.356837 3308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 13:42:58.356986 3308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:58.357084 3308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:58.382126 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.357366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52501 (local address 127.1.237.254:35815)
0927 13:42:58.357533 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.357539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.359199 (+ 1660us) server_negotiation.cc:407] Connection header received
0927 13:42:58.361974 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.361981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.362055 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.362120 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:58.364350 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.365892 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.368432 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.369213 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.369362 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.374270 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:58.374280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:58.374288 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:58.374337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:58.378256 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.379281 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.379290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.379296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.379461 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.379977 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.379983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.379986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.380381 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:58.380530 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:58.381382 (+ 852us) server_negotiation.cc:299] Negotiation successful
0927 13:42:58.381644 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:42:58.382951 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.358069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.194:52501)
0927 13:42:58.358949 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.359057 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.361658 (+ 2601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.362579 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.362589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.363311 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:58.364158 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.364183 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.366049 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.366054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.368233 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.368247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.369617 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.369623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.369872 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.370515 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:58.370545 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:58.373958 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:58.378434 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.378444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.378460 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.379140 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.379591 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.379597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.379602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.379860 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.380515 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:58.380523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:58.380996 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 13:42:58.381275 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 13:42:58.381856 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260927 13:42:58.398715 3308 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:46021
I20260927 13:42:58.398749 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:46021 every 8 connection(s)
I20260927 13:42:58.399797 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:42:58.401526 3443 heartbeater.cc:347] Connected to a master server at 127.1.237.254:35815
I20260927 13:42:58.401770 3443 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:58.402343 3443 heartbeater.cc:511] Master 127.1.237.254:35815 requested a full tablet report, sending...
I20260927 13:42:58.403610 3096 ts_manager.cc:194] Registered new tserver with Master: c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021)
I20260927 13:42:58.403852 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.389954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:56317 (local address 127.1.237.254:35815)
0927 13:42:58.390104 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.390111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.391014 (+ 903us) server_negotiation.cc:407] Connection header received
0927 13:42:58.391215 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.391223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.391315 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.391441 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:58.392691 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.393945 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.395821 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.396257 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.396385 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.398403 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:58.398409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:58.398416 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:58.398449 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:58.401283 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.401953 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.401962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.401964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.402076 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.402409 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.402416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.402419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.402712 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:58.402865 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:58.403257 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 13:42:58.403471 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 13:42:58.404095 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.390000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.194:56317)
0927 13:42:58.390894 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.390920 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.391065 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.391444 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.391463 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.392046 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:58.392463 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.392492 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.394094 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.394101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.395677 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.395687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.396438 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.396445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.396651 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.397261 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:58.397279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:58.398185 (+ 906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:58.401397 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.401403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.401406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.401852 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.402179 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.402183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.402187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.402309 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.402839 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:58.402843 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:58.403006 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 13:42:58.403140 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 13:42:58.403413 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":584,"threads_started":1}
I20260927 13:42:58.404685 3096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:52501
I20260927 13:42:58.405864 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3308
I20260927 13:42:58.406049 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:42:58.406447 1975 external_mini_cluster.cc:1428] Setting key 360644172211337e301c8ec65036aaa0
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:42:58.442162 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:35815
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:42:58.740042 3447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:42:58.740276 3447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:42:58.740435 3447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:42:58.747232 3447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:42:58.747336 3447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:42:58.747481 3447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:42:58.747598 3447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:42:58.756270 3447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:42:58.757308 3447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:42:58.758553 3447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:42:58.770859 3452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:58.770915 3453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:42:58.770941 3455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:42:58.776382 3447 server_base.cc:1034] running on GCE node
I20260927 13:42:58.776963 3447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:42:58.777786 3447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:42:58.779029 3447 hybrid_clock.cc:648] HybridClock initialized: now 1790516578778957 us; error 36 us; skew 500 ppm
I20260927 13:42:58.779342 3447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:42:58.786648 3447 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:42:58.788015 3447 webserver.cc:460] Webserver started at http://127.1.237.195:33409/ using document root <none> and password file <none>
I20260927 13:42:58.788679 3447 fs_manager.cc:359] Metadata directory not provided
I20260927 13:42:58.788797 3447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:42:58.789099 3447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:42:58.791870 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "99755b5b18d54600b515e86a0307c1f7"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "6708c3f9454aa38db439bfeb035cae30"
server_key_iv: "7efe0fd73ee1a2074c8a2c98313a65aa"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.792629 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "99755b5b18d54600b515e86a0307c1f7"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "6708c3f9454aa38db439bfeb035cae30"
server_key_iv: "7efe0fd73ee1a2074c8a2c98313a65aa"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.797526 3447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:42:58.800693 3462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:58.801846 3447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:42:58.802000 3447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "99755b5b18d54600b515e86a0307c1f7"
format_stamp: "Formatted at 2026-09-27 13:42:58 on dist-test-slave-7g3l"
server_key: "6708c3f9454aa38db439bfeb035cae30"
server_key_iv: "7efe0fd73ee1a2074c8a2c98313a65aa"
server_key_version: "encryptionkey@0"
I20260927 13:42:58.802212 3447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:42:58.855638 3447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:42:58.859543 3447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:42:58.859804 3447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:42:58.862203 3447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:42:58.862301 3447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:42:58.862439 3447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:42:58.862511 3447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:58.883574 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.862923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:48379 (local address 127.1.237.254:35815)
0927 13:42:58.863079 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.863084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.864498 (+ 1414us) server_negotiation.cc:407] Connection header received
0927 13:42:58.867284 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.867291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.867367 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.867473 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:58.869474 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.870711 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.872804 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.873572 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.873730 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.877778 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:58.877784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:58.877787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:58.877818 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:58.880542 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.881442 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.881449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.881452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.881566 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.881936 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.881941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.881944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.882171 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:58.882350 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:58.883006 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 13:42:58.883249 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:42:58.884526 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.863488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.195:48379)
0927 13:42:58.864304 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.864376 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.866931 (+ 2555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.867748 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.867761 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.868472 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:58.869250 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.869271 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.870873 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.870880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.872640 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.872660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.873816 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.873826 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.874047 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.874600 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:58.874623 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:58.877566 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:58.880661 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.880668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.880684 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.881333 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.881668 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.881673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.881677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.881836 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.882313 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:58.882319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:58.882634 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 13:42:58.882878 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:42:58.883346 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":308,"threads_started":1}
I20260927 13:42:58.905742 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.889792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.1.237.195:44613)
0927 13:42:58.890643 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.890680 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.890810 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.891358 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.891361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.892002 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:58.892511 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.892529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.893777 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.893780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.895782 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.895793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.896975 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.896980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.897158 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.897900 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:58.897928 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:58.899095 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:58.903288 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.903295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.903312 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.903776 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.904134 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:58.904138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:58.904145 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:58.904255 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:58.904731 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:58.904736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:58.904899 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 13:42:58.905116 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 13:42:58.905390 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":490,"threads_started":1}
I20260927 13:42:58.906018 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.889658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:44613 (local address 127.1.237.254:35815)
0927 13:42:58.889796 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.889799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.890745 (+ 946us) server_negotiation.cc:407] Connection header received
0927 13:42:58.890989 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.890996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.891078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.891170 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:58.892712 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.893623 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.895913 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.896620 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.896769 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.899314 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:58.899323 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:58.899330 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:58.899375 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:58.903130 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.903899 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.903905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.903909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.904027 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.904363 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.904369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.904372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.904602 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:58.904742 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:58.905248 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 13:42:58.905535 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 13:42:58.919226 3447 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:43549
I20260927 13:42:58.919255 3581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:43549 every 8 connection(s)
I20260927 13:42:58.920367 3447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:42:58.922173 3582 heartbeater.cc:347] Connected to a master server at 127.1.237.254:35815
I20260927 13:42:58.922367 3582 heartbeater.cc:464] Registering TS with master...
I20260927 13:42:58.922922 3582 heartbeater.cc:511] Master 127.1.237.254:35815 requested a full tablet report, sending...
I20260927 13:42:58.924125 3097 ts_manager.cc:194] Registered new tserver with Master: 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:58.924880 3097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:48379
I20260927 13:42:58.926087 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3447
I20260927 13:42:58.926268 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:42:58.926793 1975 external_mini_cluster.cc:1428] Setting key 4d22e9d36f6089a79e1395c12976841a
I20260927 13:42:58.929345 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:42:58.943131 3304 heartbeater.cc:503] Master 127.1.237.254:35815 was elected leader, sending a full tablet report...
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:42:58 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:58.970556 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.958440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.1.237.254:35815)
0927 13:42:58.958659 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.958669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.958961 (+ 292us) server_negotiation.cc:407] Connection header received
0927 13:42:58.959090 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.959094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.959193 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.959270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:58.960522 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.961225 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.962377 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.962752 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.962830 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:58.966381 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:58.966386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:58.966389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:58.966416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:58.968670 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.969192 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.969196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.969198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.969299 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:58.969572 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:58.969575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:58.969577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:58.969810 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:58.969963 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:58.970173 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 13:42:58.970315 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:42:58.974015 3097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:42:58.980434 3097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:42:59.001370 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.991159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:46021 (local address 127.0.0.1:46058)
0927 13:42:58.991815 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.991857 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.992083 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.995081 (+ 2998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.995085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.995112 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:58.995521 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.995535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.997274 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.997280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.999036 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.999049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.000385 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.000391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.000562 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.000619 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.000750 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.000939 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":3,"thread_start_us":489,"threads_started":1}
I20260927 13:42:59.001608 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.993084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.1.237.193:44837)
0927 13:42:58.993708 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.993719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.993751 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:42:58.993840 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.993846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.994088 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.994329 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:58.995016 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.996406 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.999077 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.000578 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.000710 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.000894 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.001144 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.001302 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":326,"threads_started":1}
I20260927 13:42:59.001811 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.992502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44837 (local address 127.0.0.1:40296)
0927 13:42:58.992999 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.993014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.993118 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.994337 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.994344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.994372 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:58.994789 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.994803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.996512 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.996520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.998856 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.998871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.000768 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.000773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.000950 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.000976 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.001062 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.001182 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":4,"thread_start_us":777,"threads_started":1}
I20260927 13:42:59.002014 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.991523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.1.237.194:46021)
0927 13:42:58.994343 (+ 2820us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.994350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.994374 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:42:58.994481 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.994490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.994696 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.994880 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:58.995743 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.997125 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.999219 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.000213 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.000359 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.000545 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.001612 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.001742 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2687,"thread_start_us":2542,"threads_started":1}
I20260927 13:42:59.002113 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.991792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:43549 (local address 127.0.0.1:55360)
0927 13:42:58.992406 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:58.992422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:58.992527 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:58.994720 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:58.994728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:58.994754 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:58.995204 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.995219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.997170 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.997175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:58.999375 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:58.999389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.001200 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.001206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.001365 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.001398 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.001700 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.001824 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":936,"threads_started":1}
I20260927 13:42:59.003046 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:58.992464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.1.237.195:43549)
0927 13:42:58.994171 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:58.994178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:58.994205 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:42:58.994293 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:58.994299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:58.994534 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:58.994719 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:58.995429 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:58.997029 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:58.999568 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.000967 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.002378 (+ 1411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.002561 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.002618 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.002754 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1519,"thread_start_us":1302,"threads_started":1}
I20260927 13:42:59.004910 3234 tablet_service.cc:1467] Processing CreateTablet for tablet cd1148496dc5401e845599ee5a4f9a53 (DEFAULT_TABLE table=test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.004909 3373 tablet_service.cc:1467] Processing CreateTablet for tablet cd1148496dc5401e845599ee5a4f9a53 (DEFAULT_TABLE table=test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.006063 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1148496dc5401e845599ee5a4f9a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.006076 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1148496dc5401e845599ee5a4f9a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.008216 3512 tablet_service.cc:1467] Processing CreateTablet for tablet cd1148496dc5401e845599ee5a4f9a53 (DEFAULT_TABLE table=test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.009286 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1148496dc5401e845599ee5a4f9a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.016530 3601 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Bootstrap starting.
I20260927 13:42:59.019459 3601 tablet_bootstrap.cc:654] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.020402 3601 log.cc:824] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:59.023612 3601 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: No bootstrap required, opened a new log
I20260927 13:42:59.023840 3601 ts_tablet_manager.cc:1397] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:42:59.025959 3604 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Bootstrap starting.
I20260927 13:42:59.026151 3602 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Bootstrap starting.
I20260927 13:42:59.029109 3602 tablet_bootstrap.cc:654] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.029452 3604 tablet_bootstrap.cc:654] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.030321 3602 log.cc:824] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:59.030594 3604 log.cc:824] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:42:59.033017 3604 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: No bootstrap required, opened a new log
I20260927 13:42:59.033272 3604 ts_tablet_manager.cc:1397] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 13:42:59.033565 3602 tablet_bootstrap.cc:492] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: No bootstrap required, opened a new log
I20260927 13:42:59.033798 3602 ts_tablet_manager.cc:1397] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260927 13:42:59.033967 3601 raft_consensus.cc:348] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.034379 3601 raft_consensus.cc:374] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.034497 3601 raft_consensus.cc:729] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296e5d8884f46f48ad9f669892684b8, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.035044 3601 consensus_queue.cc:260] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.036549 3601 ts_tablet_manager.cc:1428] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 13:42:59.041833 3604 raft_consensus.cc:348] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.042148 3604 raft_consensus.cc:374] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.042258 3604 raft_consensus.cc:729] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4228715b0894a5687f3122fa42872ec, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.042814 3604 consensus_queue.cc:260] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.042866 3602 raft_consensus.cc:348] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.043257 3602 raft_consensus.cc:374] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.043371 3602 raft_consensus.cc:729] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99755b5b18d54600b515e86a0307c1f7, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.043761 3443 heartbeater.cc:503] Master 127.1.237.254:35815 was elected leader, sending a full tablet report...
I20260927 13:42:59.043913 3602 consensus_queue.cc:260] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.044843 3582 heartbeater.cc:503] Master 127.1.237.254:35815 was elected leader, sending a full tablet report...
I20260927 13:42:59.045290 3604 ts_tablet_manager.cc:1428] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260927 13:42:59.045490 3602 ts_tablet_manager.cc:1428] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 13:42:59.048019 3608 raft_consensus.cc:482] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:42:59.048260 3608 raft_consensus.cc:504] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.049746 3608 leader_election.cc:288] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193:44837), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.058346 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.050656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:50279 (local address 127.1.237.193:44837)
0927 13:42:59.050831 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.050837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.050938 (+ 101us) server_negotiation.cc:407] Connection header received
0927 13:42:59.051055 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.051059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.051119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.051231 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:59.051977 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.053456 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.056329 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.057582 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.057756 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.057838 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.058102 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.058186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:42:59.058416 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.050186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44837 (local address 127.1.237.194:50279)
0927 13:42:59.050850 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.050865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.050966 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.051241 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.051246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.051269 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:59.051768 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.051786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.053624 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.053630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.056163 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.056176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.057771 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.057807 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.057976 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.058006 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.058081 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.058162 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":165,"thread_start_us":492,"threads_started":1}
I20260927 13:42:59.059449 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.049979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:43549 (local address 127.1.237.194:58793)
0927 13:42:59.050667 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.050692 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.050883 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.051259 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.051263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.051278 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:59.051879 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.051891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.053626 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.053630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.056166 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.056176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.058667 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.058673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.058862 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.058898 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.058988 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.059105 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":144,"spinlock_wait_cycles":24320,"thread_start_us":306,"threads_started":1}
I20260927 13:42:59.059490 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.050219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:58793 (local address 127.1.237.195:43549)
0927 13:42:59.050699 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.050706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.050776 (+ 70us) server_negotiation.cc:407] Connection header received
0927 13:42:59.051026 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.051031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.051107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.051256 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:59.052006 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.053480 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.056350 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.057576 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.057757 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.057838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.059102 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.059244 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260927 13:42:59.059748 3258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1148496dc5401e845599ee5a4f9a53" candidate_uuid: "c4228715b0894a5687f3122fa42872ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296e5d8884f46f48ad9f669892684b8" is_pre_election: true
I20260927 13:42:59.060161 3258 raft_consensus.cc:2454] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4228715b0894a5687f3122fa42872ec in term 0.
I20260927 13:42:59.060784 3327 leader_election.cc:302] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296e5d8884f46f48ad9f669892684b8, c4228715b0894a5687f3122fa42872ec; no voters:
I20260927 13:42:59.061156 3608 raft_consensus.cc:2781] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:42:59.061321 3608 raft_consensus.cc:482] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:42:59.061206 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1148496dc5401e845599ee5a4f9a53" candidate_uuid: "c4228715b0894a5687f3122fa42872ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7" is_pre_election: true
I20260927 13:42:59.061429 3608 raft_consensus.cc:3037] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.061676 3537 raft_consensus.cc:2454] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4228715b0894a5687f3122fa42872ec in term 0.
I20260927 13:42:59.063859 3608 raft_consensus.cc:504] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.064368 3608 leader_election.cc:288] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [CANDIDATE]: Term 1 election: Requested vote from peers 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193:44837), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.064752 3258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1148496dc5401e845599ee5a4f9a53" candidate_uuid: "c4228715b0894a5687f3122fa42872ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296e5d8884f46f48ad9f669892684b8"
I20260927 13:42:59.064752 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1148496dc5401e845599ee5a4f9a53" candidate_uuid: "c4228715b0894a5687f3122fa42872ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7"
I20260927 13:42:59.064935 3537 raft_consensus.cc:3037] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.064935 3258 raft_consensus.cc:3037] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.068454 3258 raft_consensus.cc:2454] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4228715b0894a5687f3122fa42872ec in term 1.
I20260927 13:42:59.068454 3537 raft_consensus.cc:2454] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4228715b0894a5687f3122fa42872ec in term 1.
I20260927 13:42:59.068842 3326 leader_election.cc:302] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99755b5b18d54600b515e86a0307c1f7, c4228715b0894a5687f3122fa42872ec; no voters:
I20260927 13:42:59.069154 3608 raft_consensus.cc:2781] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:59.069568 3608 raft_consensus.cc:686] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 1 LEADER]: Becoming Leader. State: Replica: c4228715b0894a5687f3122fa42872ec, State: Running, Role: LEADER
I20260927 13:42:59.069991 3608 consensus_queue.cc:237] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.074103 3096 catalog_manager.cc:5495] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec reported cstate change: term changed from 0 to 1, leader changed from <none> to c4228715b0894a5687f3122fa42872ec (127.1.237.194). New cstate: current_term: 1 leader_uuid: "c4228715b0894a5687f3122fa42872ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20260927 13:42:59.104636 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.099395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.1.237.194:46021)
0927 13:42:59.099550 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.099555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.099721 (+ 166us) server_negotiation.cc:407] Connection header received
0927 13:42:59.099964 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.099968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.100037 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.100154 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:42:59.100407 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.101275 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.102435 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.103261 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.103352 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.103755 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.104218 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.104314 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.104398 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.104453 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 13:42:59.111773 3258 raft_consensus.cc:1261] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Refusing update from remote peer c4228715b0894a5687f3122fa42872ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.111780 3537 raft_consensus.cc:1261] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Refusing update from remote peer c4228715b0894a5687f3122fa42872ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.112650 3612 consensus_queue.cc:1035] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.112937 3608 consensus_queue.cc:1035] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.123461 3615 mvcc.cc:205] Tried to move back new op lower bound from 7333955908033667072 to 7333955907873849344. Current Snapshot: MvccSnapshot[applied={T|T < 7333955908033667072}]
I20260927 13:42:59.130182 3616 mvcc.cc:205] Tried to move back new op lower bound from 7333955908033667072 to 7333955907873849344. Current Snapshot: MvccSnapshot[applied={T|T < 7333955908033667072}]
I20260927 13:42:59.140743 3096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:42:59.140980 3096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:42:59.144366 3096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Sending DeleteTablet for 3 replicas of tablet cd1148496dc5401e845599ee5a4f9a53
I20260927 13:42:59.145418 3373 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1148496dc5401e845599ee5a4f9a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:46058
I20260927 13:42:59.145462 3234 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1148496dc5401e845599ee5a4f9a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:40296
I20260927 13:42:59.145692 3512 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1148496dc5401e845599ee5a4f9a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:55360
I20260927 13:42:59.146565 3096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:42:59.147524 3096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:42:59.150403 3627 tablet_replica.cc:330] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: stopping tablet replica
I20260927 13:42:59.150848 3627 raft_consensus.cc:2229] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:42:59.152913 3627 raft_consensus.cc:2258] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.155354 3511 tablet_service.cc:1467] Processing CreateTablet for tablet b2549027ebb542dfa8f59aa7072e8432 (DEFAULT_TABLE table=test-table [id=5b384fda9de7438db9221f6fb00f79e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.155551 3372 tablet_service.cc:1467] Processing CreateTablet for tablet b2549027ebb542dfa8f59aa7072e8432 (DEFAULT_TABLE table=test-table [id=5b384fda9de7438db9221f6fb00f79e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.155653 3233 tablet_service.cc:1467] Processing CreateTablet for tablet b2549027ebb542dfa8f59aa7072e8432 (DEFAULT_TABLE table=test-table [id=5b384fda9de7438db9221f6fb00f79e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:42:59.155776 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2549027ebb542dfa8f59aa7072e8432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.156160 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2549027ebb542dfa8f59aa7072e8432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.156175 3233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2549027ebb542dfa8f59aa7072e8432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.161991 3602 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Bootstrap starting.
I20260927 13:42:59.163079 3601 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Bootstrap starting.
I20260927 13:42:59.163892 3602 tablet_bootstrap.cc:654] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.164484 3604 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Bootstrap starting.
I20260927 13:42:59.164589 3601 tablet_bootstrap.cc:654] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.166584 3604 tablet_bootstrap.cc:654] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.173239 3627 ts_tablet_manager.cc:1905] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:42:59.174270 3625 tablet_replica.cc:330] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: stopping tablet replica
I20260927 13:42:59.175187 3625 raft_consensus.cc:2229] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:42:59.176319 3625 raft_consensus.cc:2258] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.180881 3626 tablet_replica.cc:330] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: stopping tablet replica
I20260927 13:42:59.181243 3626 raft_consensus.cc:2229] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:42:59.181782 3626 raft_consensus.cc:2258] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.182905 3602 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: No bootstrap required, opened a new log
I20260927 13:42:59.183085 3602 ts_tablet_manager.cc:1397] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20260927 13:42:59.183112 3601 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: No bootstrap required, opened a new log
I20260927 13:42:59.183255 3601 ts_tablet_manager.cc:1397] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20260927 13:42:59.183804 3626 ts_tablet_manager.cc:1905] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:42:59.183895 3602 raft_consensus.cc:348] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.184173 3602 raft_consensus.cc:374] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.184007 3601 raft_consensus.cc:348] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.184295 3602 raft_consensus.cc:729] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99755b5b18d54600b515e86a0307c1f7, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.184306 3601 raft_consensus.cc:374] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.184427 3601 raft_consensus.cc:729] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296e5d8884f46f48ad9f669892684b8, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.184600 3602 consensus_queue.cc:260] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.184780 3601 consensus_queue.cc:260] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.185289 3604 tablet_bootstrap.cc:492] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: No bootstrap required, opened a new log
I20260927 13:42:59.185557 3604 ts_tablet_manager.cc:1397] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20260927 13:42:59.186484 3604 raft_consensus.cc:348] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.186796 3604 raft_consensus.cc:374] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.186931 3604 raft_consensus.cc:729] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4228715b0894a5687f3122fa42872ec, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.187387 3602 ts_tablet_manager.cc:1428] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 13:42:59.187296 3604 consensus_queue.cc:260] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.187804 3604 ts_tablet_manager.cc:1428] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:42:59.179123 3625 ts_tablet_manager.cc:1905] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:42:59.188130 3601 ts_tablet_manager.cc:1428] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260927 13:42:59.192292 3626 ts_tablet_manager.cc:1918] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:42:59.192521 3626 log.cc:1196] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/cd1148496dc5401e845599ee5a4f9a53
I20260927 13:42:59.193106 3626 ts_tablet_manager.cc:1939] T cd1148496dc5401e845599ee5a4f9a53 P 1296e5d8884f46f48ad9f669892684b8: Deleting consensus metadata
I20260927 13:42:59.194358 3627 ts_tablet_manager.cc:1918] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:42:59.194516 3627 log.cc:1196] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/cd1148496dc5401e845599ee5a4f9a53
I20260927 13:42:59.194850 3084 catalog_manager.cc:4841] TS 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193:44837): tablet cd1148496dc5401e845599ee5a4f9a53 (table test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]) successfully deleted
I20260927 13:42:59.195076 3627 ts_tablet_manager.cc:1939] T cd1148496dc5401e845599ee5a4f9a53 P 99755b5b18d54600b515e86a0307c1f7: Deleting consensus metadata
I20260927 13:42:59.196358 3083 catalog_manager.cc:4841] TS 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549): tablet cd1148496dc5401e845599ee5a4f9a53 (table test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]) successfully deleted
I20260927 13:42:59.198585 3625 ts_tablet_manager.cc:1918] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:42:59.198753 3625 log.cc:1196] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/cd1148496dc5401e845599ee5a4f9a53
I20260927 13:42:59.199234 3625 ts_tablet_manager.cc:1939] T cd1148496dc5401e845599ee5a4f9a53 P c4228715b0894a5687f3122fa42872ec: Deleting consensus metadata
I20260927 13:42:59.200645 3084 catalog_manager.cc:4841] TS c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021): tablet cd1148496dc5401e845599ee5a4f9a53 (table test-table [id=8ebc4eacb48b4ec3a567b6e161edc6bb]) successfully deleted
I20260927 13:42:59.322729 3607 raft_consensus.cc:482] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:42:59.322908 3607 raft_consensus.cc:504] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.324877 3607 leader_election.cc:288] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.332821 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.325355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:45225 (local address 127.1.237.194:46021)
0927 13:42:59.325594 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.325603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.326368 (+ 765us) server_negotiation.cc:407] Connection header received
0927 13:42:59.326483 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.326488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.326592 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.326684 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:59.327448 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.328132 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.330748 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.331800 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.331966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.332054 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.332375 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.332501 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:42:59.332830 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.325255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:46021 (local address 127.1.237.193:45225)
0927 13:42:59.326257 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.326271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.326380 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.326800 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.326804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.326821 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:59.327298 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.327312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.328251 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.328259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.330491 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.330517 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.332001 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.332015 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.332212 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.332251 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.332344 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.332462 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":43,"thread_start_us":477,"threads_started":1}
I20260927 13:42:59.332964 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.325494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:38475 (local address 127.1.237.195:43549)
0927 13:42:59.325670 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.325675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.326144 (+ 469us) server_negotiation.cc:407] Connection header received
0927 13:42:59.326455 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.326463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.326533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.326617 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:42:59.327272 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.328398 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.330775 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.332044 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.332196 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.332262 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.332578 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.332702 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:42:59.333814 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.325169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:43549 (local address 127.1.237.193:38475)
0927 13:42:59.326033 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.326052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.326249 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.326646 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.326651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.326673 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:42:59.327069 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.327081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.328544 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.328549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.330590 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.330604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.332239 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.332244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.332399 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.332427 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.333110 (+ 683us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.333211 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":24,"thread_start_us":860,"threads_started":1}
I20260927 13:42:59.334434 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2549027ebb542dfa8f59aa7072e8432" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7" is_pre_election: true
I20260927 13:42:59.334314 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2549027ebb542dfa8f59aa7072e8432" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228715b0894a5687f3122fa42872ec" is_pre_election: true
I20260927 13:42:59.334663 3537 raft_consensus.cc:2454] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 0.
I20260927 13:42:59.334745 3393 raft_consensus.cc:2454] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 0.
I20260927 13:42:59.335196 3186 leader_election.cc:302] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296e5d8884f46f48ad9f669892684b8, 99755b5b18d54600b515e86a0307c1f7; no voters:
I20260927 13:42:59.335577 3607 raft_consensus.cc:2781] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:42:59.335683 3607 raft_consensus.cc:482] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:42:59.335786 3607 raft_consensus.cc:3037] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.337762 3607 raft_consensus.cc:504] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.338239 3607 leader_election.cc:288] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 election: Requested vote from peers c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.338619 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2549027ebb542dfa8f59aa7072e8432" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228715b0894a5687f3122fa42872ec"
I20260927 13:42:59.338673 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2549027ebb542dfa8f59aa7072e8432" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7"
I20260927 13:42:59.338807 3393 raft_consensus.cc:3037] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.338893 3537 raft_consensus.cc:3037] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.340894 3537 raft_consensus.cc:2454] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 1.
I20260927 13:42:59.341313 3186 leader_election.cc:302] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296e5d8884f46f48ad9f669892684b8, 99755b5b18d54600b515e86a0307c1f7; no voters:
I20260927 13:42:59.341584 3607 raft_consensus.cc:2781] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:59.342041 3393 raft_consensus.cc:2454] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 1.
I20260927 13:42:59.342051 3607 raft_consensus.cc:686] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1296e5d8884f46f48ad9f669892684b8, State: Running, Role: LEADER
I20260927 13:42:59.342417 3607 consensus_queue.cc:237] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.346091 3095 catalog_manager.cc:5495] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } health_report { overall_health: UNKNOWN } } }
W20260927 13:42:59.402969 3444 tablet.cc:2369] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:42:59 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:42:59.413627 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.397290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:35815 (local address 127.0.0.1:38650)
0927 13:42:59.397897 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.397917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.398054 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.398440 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.398447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.399352 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:42:59.399692 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.399705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.401044 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.401050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.402491 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.402505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.403454 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.403460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.403654 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.404348 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:42:59.404369 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:42:59.407343 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:42:59.410825 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:59.410837 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:59.410843 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:59.411391 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:59.411841 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:42:59.411846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:42:59.411850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:42:59.412023 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:42:59.412580 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:42:59.412596 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:42:59.412769 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.412944 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.413198 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":300,"threads_started":1}
I20260927 13:42:59.413666 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.397261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.1.237.254:35815)
0927 13:42:59.397420 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.397429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.398000 (+ 571us) server_negotiation.cc:407] Connection header received
0927 13:42:59.398207 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.398213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.398296 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.398396 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.399856 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.400907 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.402634 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.403288 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.403376 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.407529 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.407536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.407541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.407573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.410685 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.411576 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.411581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.411585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.411731 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.412148 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.412153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.412157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.412453 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.412581 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.413061 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.413336 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":38}
I20260927 13:42:59.419360 3095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:38650:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:42:59.420480 3095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 13:42:59.421914 3583 tablet.cc:2369] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:42:59.427737 3234 tablet_service.cc:1467] Processing CreateTablet for tablet e1371dc6b72941ebae4be082ac226e90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b13cce2502942e2a5c93565a5fd5598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:42:59.427807 3372 tablet_service.cc:1467] Processing CreateTablet for tablet e1371dc6b72941ebae4be082ac226e90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b13cce2502942e2a5c93565a5fd5598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:42:59.428243 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1371dc6b72941ebae4be082ac226e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.428273 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1371dc6b72941ebae4be082ac226e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.430028 3511 tablet_service.cc:1467] Processing CreateTablet for tablet e1371dc6b72941ebae4be082ac226e90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b13cce2502942e2a5c93565a5fd5598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:42:59.430485 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1371dc6b72941ebae4be082ac226e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:42:59.435122 3604 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: Bootstrap starting.
I20260927 13:42:59.435604 3601 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: Bootstrap starting.
I20260927 13:42:59.436960 3604 tablet_bootstrap.cc:654] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.437175 3601 tablet_bootstrap.cc:654] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: Neither blocks nor log segments found. Creating new log.
W20260927 13:42:59.438520 3305 tablet.cc:2369] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:42:59.447592 3602 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: Bootstrap starting.
I20260927 13:42:59.449055 3602 tablet_bootstrap.cc:654] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: Neither blocks nor log segments found. Creating new log.
I20260927 13:42:59.451059 3601 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: No bootstrap required, opened a new log
I20260927 13:42:59.451450 3601 ts_tablet_manager.cc:1397] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260927 13:42:59.452203 3601 raft_consensus.cc:348] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.452510 3601 raft_consensus.cc:374] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.452684 3601 raft_consensus.cc:729] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296e5d8884f46f48ad9f669892684b8, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.453042 3601 consensus_queue.cc:260] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.453608 3607 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.453796 3601 ts_tablet_manager.cc:1428] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:42:59.453979 3607 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.460176 3602 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: No bootstrap required, opened a new log
I20260927 13:42:59.460847 3602 ts_tablet_manager.cc:1397] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20260927 13:42:59.461745 3602 raft_consensus.cc:348] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.462087 3602 raft_consensus.cc:374] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.462216 3602 raft_consensus.cc:729] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99755b5b18d54600b515e86a0307c1f7, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.462638 3602 consensus_queue.cc:260] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.462823 3604 tablet_bootstrap.cc:492] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: No bootstrap required, opened a new log
I20260927 13:42:59.463042 3604 ts_tablet_manager.cc:1397] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20260927 13:42:59.463907 3604 raft_consensus.cc:348] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.464203 3604 raft_consensus.cc:374] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:42:59.464319 3604 raft_consensus.cc:729] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4228715b0894a5687f3122fa42872ec, State: Initialized, Role: FOLLOWER
I20260927 13:42:59.464627 3604 consensus_queue.cc:260] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.465118 3622 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.465440 3622 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.466403 3602 ts_tablet_manager.cc:1428] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:42:59.464103 3609 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.466624 3609 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.467011 3604 ts_tablet_manager.cc:1428] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 13:42:59.667410 3607 raft_consensus.cc:482] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:42:59.667678 3607 raft_consensus.cc:504] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.668229 3607 leader_election.cc:288] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.668751 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1371dc6b72941ebae4be082ac226e90" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228715b0894a5687f3122fa42872ec" is_pre_election: true
I20260927 13:42:59.668783 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1371dc6b72941ebae4be082ac226e90" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7" is_pre_election: true
I20260927 13:42:59.668969 3393 raft_consensus.cc:2454] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 0.
I20260927 13:42:59.669023 3537 raft_consensus.cc:2454] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 0.
I20260927 13:42:59.669394 3187 leader_election.cc:302] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296e5d8884f46f48ad9f669892684b8, c4228715b0894a5687f3122fa42872ec; no voters:
I20260927 13:42:59.669659 3607 raft_consensus.cc:2781] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:42:59.669734 3607 raft_consensus.cc:482] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:42:59.669854 3607 raft_consensus.cc:3037] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.671725 3607 raft_consensus.cc:504] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.672616 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1371dc6b72941ebae4be082ac226e90" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4228715b0894a5687f3122fa42872ec"
I20260927 13:42:59.672711 3607 leader_election.cc:288] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 election: Requested vote from peers c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021), 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549)
I20260927 13:42:59.672811 3393 raft_consensus.cc:3037] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.673043 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1371dc6b72941ebae4be082ac226e90" candidate_uuid: "1296e5d8884f46f48ad9f669892684b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99755b5b18d54600b515e86a0307c1f7"
I20260927 13:42:59.673192 3537 raft_consensus.cc:3037] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:42:59.674541 3393 raft_consensus.cc:2454] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 1.
I20260927 13:42:59.674861 3187 leader_election.cc:302] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296e5d8884f46f48ad9f669892684b8, c4228715b0894a5687f3122fa42872ec; no voters:
I20260927 13:42:59.674907 3537 raft_consensus.cc:2454] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1296e5d8884f46f48ad9f669892684b8 in term 1.
I20260927 13:42:59.675115 3607 raft_consensus.cc:2781] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:42:59.675309 3607 raft_consensus.cc:686] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1296e5d8884f46f48ad9f669892684b8, State: Running, Role: LEADER
I20260927 13:42:59.675561 3607 consensus_queue.cc:237] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } }
I20260927 13:42:59.676963 3633 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: TxnStatusTablet state changed. Reason: New leader 1296e5d8884f46f48ad9f669892684b8. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.677274 3633 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:42:59.677881 3645 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:42:59.678103 3645 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:42:59.678305 3096 catalog_manager.cc:5495] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20260927 13:42:59.705868 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.699738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44837 (local address 127.0.0.1:40298)
0927 13:42:59.699952 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.699975 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.700112 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.700641 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.700646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.700660 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:42:59.700854 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.700863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.701866 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.701873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.703511 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.703528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.704216 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.704222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.704426 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.704490 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.705313 (+ 823us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.705332 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.705420 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.705563 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 13:42:59.705891 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.699772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.1.237.193:44837)
0927 13:42:59.700296 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.700301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.700314 (+ 13us) server_negotiation.cc:407] Connection header received
0927 13:42:59.700379 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.700382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.700452 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.700599 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:42:59.700979 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.701702 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.703645 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.704060 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.704159 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.704660 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.705195 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.705311 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.705473 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.705538 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
I20260927 13:42:59.709779 3537 raft_consensus.cc:1261] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Refusing update from remote peer 1296e5d8884f46f48ad9f669892684b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.709926 3393 raft_consensus.cc:1261] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Refusing update from remote peer 1296e5d8884f46f48ad9f669892684b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.710536 3633 consensus_queue.cc:1035] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.710850 3607 consensus_queue.cc:1035] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.723486 3622 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: TxnStatusTablet state changed. Reason: New leader 1296e5d8884f46f48ad9f669892684b8. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.723749 3622 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.724704 3609 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: TxnStatusTablet state changed. Reason: New leader 1296e5d8884f46f48ad9f669892684b8. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.724937 3609 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.729301 3607 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.729592 3607 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:42:59.734762 3622 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.734977 3622 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P c4228715b0894a5687f3122fa42872ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.734923 3607 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.735249 3607 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 1296e5d8884f46f48ad9f669892684b8: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:42:59.737218 3609 tablet_replica.cc:439] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1296e5d8884f46f48ad9f669892684b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296e5d8884f46f48ad9f669892684b8" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44837 } } peers { permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 } } peers { permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 } } }
I20260927 13:42:59.737423 3609 tablet_replica.cc:442] T e1371dc6b72941ebae4be082ac226e90 P 99755b5b18d54600b515e86a0307c1f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:42:59.749065 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.739679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.1.237.193:44837)
0927 13:42:59.740148 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.740157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.743721 (+ 3564us) server_negotiation.cc:407] Connection header received
0927 13:42:59.743885 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.743888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.743938 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.744000 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:42:59.744446 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.745166 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.747564 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.747934 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.748021 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.748446 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.748526 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.748584 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.748729 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.748781 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260927 13:42:59.764451 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.757999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44837 (local address 127.1.237.193:36099)
0927 13:42:59.759114 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 13:42:59.759130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:42:59.759271 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:42:59.759612 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:42:59.759615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:42:59.759624 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:42:59.759887 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.759893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.761254 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.761263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.762565 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:42:59.762579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.763507 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.763511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:42:59.763611 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.763652 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.763995 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.764008 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 13:42:59.764070 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 13:42:59.764153 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":779,"threads_started":1}
I20260927 13:42:59.764477 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.757910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:36099 (local address 127.1.237.193:44837)
0927 13:42:59.758086 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.758092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.759229 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 13:42:59.759391 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.759396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.759513 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.759606 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:42:59.760007 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.761131 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.762686 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.763362 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.763474 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.763796 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:42:59.763902 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:42:59.763984 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.764105 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.764176 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:42:59.773288 3536 raft_consensus.cc:1261] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Refusing update from remote peer 1296e5d8884f46f48ad9f669892684b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.773862 3393 raft_consensus.cc:1261] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Refusing update from remote peer 1296e5d8884f46f48ad9f669892684b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:42:59.774262 3662 consensus_queue.cc:1035] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99755b5b18d54600b515e86a0307c1f7" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.774596 3633 consensus_queue.cc:1035] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4228715b0894a5687f3122fa42872ec" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:42:59.778090 3649 mvcc.cc:205] Tried to move back new op lower bound from 7333955910744297472 to 7333955908991254528. Current Snapshot: MvccSnapshot[applied={T|T < 7333955910744297472}]
I20260927 13:42:59.826876 3096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:42:59.827066 3096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:42:59.829806 3096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c6638cc03a49758b299071f57acd79: Sending DeleteTablet for 3 replicas of tablet b2549027ebb542dfa8f59aa7072e8432
I20260927 13:42:59.830468 3233 tablet_service.cc:1514] Processing DeleteTablet for tablet b2549027ebb542dfa8f59aa7072e8432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:40296
I20260927 13:42:59.830533 3372 tablet_service.cc:1514] Processing DeleteTablet for tablet b2549027ebb542dfa8f59aa7072e8432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:46058
I20260927 13:42:59.830907 3511 tablet_service.cc:1514] Processing DeleteTablet for tablet b2549027ebb542dfa8f59aa7072e8432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:42:59 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:55360
I20260927 13:42:59.838904 3668 tablet_replica.cc:330] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: stopping tablet replica
I20260927 13:42:59.839145 3668 raft_consensus.cc:2229] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:42:59.840095 3668 raft_consensus.cc:2258] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.840616 3666 tablet_replica.cc:330] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: stopping tablet replica
I20260927 13:42:59.840876 3666 raft_consensus.cc:2229] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:42:59.841153 3668 ts_tablet_manager.cc:1905] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:42:59.841447 3666 raft_consensus.cc:2258] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.842315 3666 ts_tablet_manager.cc:1905] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:42:59.849220 3668 ts_tablet_manager.cc:1918] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:42:59.849383 3668 log.cc:1196] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/b2549027ebb542dfa8f59aa7072e8432
I20260927 13:42:59.849783 3666 ts_tablet_manager.cc:1918] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:42:59.849852 3668 ts_tablet_manager.cc:1939] T b2549027ebb542dfa8f59aa7072e8432 P 99755b5b18d54600b515e86a0307c1f7: Deleting consensus metadata
I20260927 13:42:59.850180 3667 tablet_replica.cc:330] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: stopping tablet replica
I20260927 13:42:59.850333 3666 log.cc:1196] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/b2549027ebb542dfa8f59aa7072e8432
I20260927 13:42:59.850852 3666 ts_tablet_manager.cc:1939] T b2549027ebb542dfa8f59aa7072e8432 P 1296e5d8884f46f48ad9f669892684b8: Deleting consensus metadata
I20260927 13:42:59.850836 3083 catalog_manager.cc:4841] TS 99755b5b18d54600b515e86a0307c1f7 (127.1.237.195:43549): tablet b2549027ebb542dfa8f59aa7072e8432 (table test-table [id=5b384fda9de7438db9221f6fb00f79e3]) successfully deleted
I20260927 13:42:59.851104 3667 raft_consensus.cc:2229] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:42:59.851397 3667 raft_consensus.cc:2258] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:42:59.852509 3084 catalog_manager.cc:4841] TS 1296e5d8884f46f48ad9f669892684b8 (127.1.237.193:44837): tablet b2549027ebb542dfa8f59aa7072e8432 (table test-table [id=5b384fda9de7438db9221f6fb00f79e3]) successfully deleted
I20260927 13:42:59.853289 3667 ts_tablet_manager.cc:1905] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:42:59 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:42:59.860288 3667 ts_tablet_manager.cc:1918] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:42:59.860414 3667 log.cc:1196] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/b2549027ebb542dfa8f59aa7072e8432
I20260927 13:42:59.860882 3667 ts_tablet_manager.cc:1939] T b2549027ebb542dfa8f59aa7072e8432 P c4228715b0894a5687f3122fa42872ec: Deleting consensus metadata
I20260927 13:42:59.861836 3084 catalog_manager.cc:4841] TS c4228715b0894a5687f3122fa42872ec (127.1.237.194:46021): tablet b2549027ebb542dfa8f59aa7072e8432 (table test-table [id=5b384fda9de7438db9221f6fb00f79e3]) successfully deleted
I20260927 13:42:59.864921 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.840836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.1.237.193:44837)
0927 13:42:59.841015 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.841022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.848732 (+ 7710us) server_negotiation.cc:407] Connection header received
0927 13:42:59.849819 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.849827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.849908 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.849997 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.852115 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.853019 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.854281 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.854724 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.854845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.858836 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.858847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.858860 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.858886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.862373 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.862978 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.862986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.862990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.863124 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.863527 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.863531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.863535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.863849 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.864156 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.864420 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.864616 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 13:42:59.865649 3299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40314
I20260927 13:42:59.881378 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.869111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.1.237.254:35815)
0927 13:42:59.869287 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.869293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.869517 (+ 224us) server_negotiation.cc:407] Connection header received
0927 13:42:59.869627 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.869633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.869704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.869801 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.870824 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.871911 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.873045 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.873682 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.873803 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.875736 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.875743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.875748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.875782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.878764 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.879337 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.879343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.879345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.879465 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.879769 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.879774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.879778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.880077 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.880333 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.880877 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.881083 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:42:59.881896 3096 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658
I20260927 13:42:59.883761 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3168
I20260927 13:42:59.897784 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.884858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.1.237.193:44837)
0927 13:42:59.885035 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.885042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.885751 (+ 709us) server_negotiation.cc:407] Connection header received
0927 13:42:59.885860 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.885866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.885938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.886052 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.887148 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.888331 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.889439 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.890144 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.890269 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.892289 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.892296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.892301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.892333 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.895542 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.896077 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.896082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.896086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.896219 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.896514 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.896519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.896523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.896813 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.897023 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.897278 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.897474 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 13:42:59.898240 3299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324
W20260927 13:42:59.898702 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3168: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:42:59.898869 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3168
I20260927 13:42:59.916896 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3308
Sep 27 13:42:59 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:42:59.930641 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.917485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.1.237.194:46021)
0927 13:42:59.917982 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.917986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.918001 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:42:59.918047 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.918051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.918105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.918187 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.919417 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.920119 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.921293 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.921715 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.921863 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.925432 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.925443 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.925455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.925484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.928116 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.928709 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.928715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.928717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.928827 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.929151 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.929157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.929160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.929470 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.929775 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.930058 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.930262 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":294,"threads_started":1}
W20260927 13:42:59.931305 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46074
W20260927 13:42:59.931852 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3308: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:42:59.932032 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3308
I20260927 13:42:59.952450 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3447
Sep 27 13:42:59 dist-test-slave-7g3l krb5kdc[3052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:42:59.966693 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:42:59.953094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.1.237.195:43549)
0927 13:42:59.953519 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 13:42:59.953523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:42:59.953537 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:42:59.953582 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:42:59.953586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:42:59.953639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:42:59.953747 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:42:59.954816 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.955609 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.956756 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:42:59.957184 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:42:59.957307 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:42:59.961229 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:42:59.961239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:42:59.961251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:42:59.961279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:42:59.963935 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.964677 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.964685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.964687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.964816 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:42:59.965161 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:42:59.965168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:42:59.965171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:42:59.965485 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:42:59.965765 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 13:42:59.966051 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 13:42:59.966296 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
W20260927 13:42:59.967540 3577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55376
W20260927 13:42:59.968071 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3447: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:42:59.968209 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3447
I20260927 13:42:59.984133 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3065
I20260927 13:42:59.985069 3137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:00.094270 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3065
2026-09-27T13:43:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4324 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:00 dist-test-slave-7g3l krb5kdc[3679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:00 dist-test-slave-7g3l krb5kdc[3679](info): set up 2 sockets
Sep 27 13:43:00 dist-test-slave-7g3l krb5kdc[3679](info): commencing operation
krb5kdc: starting...
W20260927 13:43:01.159176 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:01 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:01.321806 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42195
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:01.620888 3692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:01.621194 3692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:01.621341 3692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:01.628275 3692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:01.628397 3692 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:43:01.628489 3692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:01.628599 3692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:01.628675 3692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:01.628758 3692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:01.637928 3692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42195
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:01.639075 3692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:01.640333 3692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:01.651114 3697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:01.651206 3698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:01.651224 3700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:01.652357 3692 server_base.cc:1034] running on GCE node
I20260927 13:43:01.653021 3692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:01.654529 3692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:01.655740 3692 hybrid_clock.cc:648] HybridClock initialized: now 1790516581655662 us; error 58 us; skew 500 ppm
I20260927 13:43:01.656014 3692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:01 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:01.658939 3692 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:01.660223 3692 webserver.cc:460] Webserver started at http://127.1.237.254:33401/ using document root <none> and password file <none>
I20260927 13:43:01.660820 3692 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:01.660899 3692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:01.661142 3692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:01.663877 3692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "861c9dc76f504ad395e2155c92588a26"
format_stamp: "Formatted at 2026-09-27 13:43:01 on dist-test-slave-7g3l"
server_key: "1fda85a8d4ebde2914624ed95a78b585"
server_key_iv: "4ad8115987f8f7cf9c6354230dfe1f0e"
server_key_version: "encryptionkey@0"
I20260927 13:43:01.664549 3692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "861c9dc76f504ad395e2155c92588a26"
format_stamp: "Formatted at 2026-09-27 13:43:01 on dist-test-slave-7g3l"
server_key: "1fda85a8d4ebde2914624ed95a78b585"
server_key_iv: "4ad8115987f8f7cf9c6354230dfe1f0e"
server_key_version: "encryptionkey@0"
I20260927 13:43:01.669309 3692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:01.672761 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:01.673965 3692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:43:01.674104 3692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "861c9dc76f504ad395e2155c92588a26"
format_stamp: "Formatted at 2026-09-27 13:43:01 on dist-test-slave-7g3l"
server_key: "1fda85a8d4ebde2914624ed95a78b585"
server_key_iv: "4ad8115987f8f7cf9c6354230dfe1f0e"
server_key_version: "encryptionkey@0"
I20260927 13:43:01.674271 3692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:01.686793 3692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:01.690711 3692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:01.690913 3692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:01.711009 3692 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:42195
I20260927 13:43:01.711046 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:42195 every 8 connection(s)
I20260927 13:43:01.712244 3692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:01.715617 3769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:01.715942 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3692
I20260927 13:43:01.716128 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:01.716740 1975 external_mini_cluster.cc:1428] Setting key 35f0af82fec1f4033e4864f370529faf
I20260927 13:43:01.728353 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Bootstrap starting.
Sep 27 13:43:01 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:01.731724 3769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:01.732698 3769 log.cc:824] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:01.735117 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: No bootstrap required, opened a new log
I20260927 13:43:01.737743 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:01.718477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.1.237.254:42195)
0927 13:43:01.719278 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:01.719288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:01.719328 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:43:01.721106 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:01.721125 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:01.721487 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:01.721783 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:01.723449 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:01.724679 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:01.726401 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:01.726976 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:01.727104 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:01.731393 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:01.731399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:01.731411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:01.731445 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:01.734254 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:01.734860 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:01.734865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:01.734870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:01.735021 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:01.735400 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:01.735406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:01.735409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:01.735868 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:01.736153 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:01.736537 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 13:43:01.736871 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":564,"thread_start_us":353,"threads_started":1}
I20260927 13:43:01.742597 3769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } }
I20260927 13:43:01.742901 3769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:01.742995 3769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861c9dc76f504ad395e2155c92588a26, State: Initialized, Role: FOLLOWER
I20260927 13:43:01.743537 3769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } }
I20260927 13:43:01.743779 3769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:01.743903 3769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:01.744069 3769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:01.746004 3769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } }
I20260927 13:43:01.746380 3769 leader_election.cc:302] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861c9dc76f504ad395e2155c92588a26; no voters:
I20260927 13:43:01.746852 3769 leader_election.cc:288] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:01.746915 3774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:01.747835 3774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [term 1 LEADER]: Becoming Leader. State: Replica: 861c9dc76f504ad395e2155c92588a26, State: Running, Role: LEADER
I20260927 13:43:01.748284 3774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } }
I20260927 13:43:01.749766 3774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861c9dc76f504ad395e2155c92588a26. Latest consensus state: current_term: 1 leader_uuid: "861c9dc76f504ad395e2155c92588a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } } }
I20260927 13:43:01.750049 3774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:01.752292 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861c9dc76f504ad395e2155c92588a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861c9dc76f504ad395e2155c92588a26" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42195 } } }
I20260927 13:43:01.752565 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:01.753615 3769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:01.760267 3778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:01.764222 3778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:01.771487 3778 catalog_manager.cc:1347] Generated new cluster ID: fe0b5e3c8a8f4f8a8c645cc0f7ad724b
I20260927 13:43:01.771596 3778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:01.789232 3778 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:01.791100 3778 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:01.797360 3778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Generated new TSK 0
I20260927 13:43:01.798130 3778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:01.846233 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:02.175091 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:02.175374 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:02.175539 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:02.182905 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:02.183017 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:02.183146 3795 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:43:02.183250 3795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:02.183346 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:02.192682 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:02.193812 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:02.195032 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:02.207278 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:02.207312 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:02.207247 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:02.208367 3795 server_base.cc:1034] running on GCE node
I20260927 13:43:02.208848 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:02.209687 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:02.210888 3795 hybrid_clock.cc:648] HybridClock initialized: now 1790516582210856 us; error 41 us; skew 500 ppm
I20260927 13:43:02.211234 3795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:02 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516582, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:02.214133 3795 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:02.215536 3795 webserver.cc:460] Webserver started at http://127.1.237.193:46613/ using document root <none> and password file <none>
I20260927 13:43:02.216125 3795 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:02.216284 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:02.216573 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:02.219288 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "ac79b63a036a4074b1550dcbb846a115"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "72e38cc62367f887d5d08204d91080e9"
server_key_iv: "c3f83b0b6264505c4fe50d6dda2f1045"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.219990 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "ac79b63a036a4074b1550dcbb846a115"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "72e38cc62367f887d5d08204d91080e9"
server_key_iv: "c3f83b0b6264505c4fe50d6dda2f1045"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.224519 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:02.227558 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:02.228708 3795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260927 13:43:02.228880 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "ac79b63a036a4074b1550dcbb846a115"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "72e38cc62367f887d5d08204d91080e9"
server_key_iv: "c3f83b0b6264505c4fe50d6dda2f1045"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.229054 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:02.240404 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:02.243976 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:02.244225 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:02.246518 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:02.246671 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:02.246829 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:02.246953 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:43:02 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516582, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:02.267313 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.246831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:33621 (local address 127.1.237.254:42195)
0927 13:43:02.247263 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:02.247268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:02.248300 (+ 1032us) server_negotiation.cc:407] Connection header received
0927 13:43:02.250271 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:02.250277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:02.250358 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:02.250464 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:02.252296 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.253320 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.255869 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.256276 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.256417 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.260532 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:02.260544 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:02.260548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:02.260577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:02.263880 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.265044 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.265048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.265051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.265178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.265640 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.265645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.265648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.265900 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:02.266058 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:02.266702 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 13:43:02.266913 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260927 13:43:02.268541 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.247350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.193:33621)
0927 13:43:02.248140 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:02.248179 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:02.250010 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:02.250666 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:02.250673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:02.251327 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:02.252140 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.252157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.253453 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.253465 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.255744 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.255755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.256498 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.256508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.256774 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.257365 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:02.257384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:02.260272 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:02.264062 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.264074 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.264095 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.264862 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.265323 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.265328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.265332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.265515 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.266032 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:02.266039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:02.266362 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 13:43:02.266609 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 13:43:02.267051 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":358,"threads_started":1}
I20260927 13:43:02.289062 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.275516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.193:46583)
0927 13:43:02.276166 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:02.276188 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:02.276334 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:02.276613 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:02.276617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:02.277079 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:02.277368 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.277376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.278316 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.278321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.279560 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.279571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.280510 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.280518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.280790 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.281601 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:02.281629 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:02.282884 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:02.286053 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.286058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.286065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.286626 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.286981 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.286987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.286990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.287116 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.287812 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:02.287821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:02.287991 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 13:43:02.288185 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 13:43:02.288513 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":357,"threads_started":1}
I20260927 13:43:02.289765 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.275395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:46583 (local address 127.1.237.254:42195)
0927 13:43:02.275878 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:02.275886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:02.276280 (+ 394us) server_negotiation.cc:407] Connection header received
0927 13:43:02.276447 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:02.276457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:02.276505 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:02.276593 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:02.277489 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.278179 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.279707 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.280306 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.280444 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.283056 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:02.283060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:02.283062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:02.283087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:02.285932 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.286739 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.286746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.286751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.286864 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.287256 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.287262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.287266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.287598 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:02.287745 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:02.289368 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0927 13:43:02.289562 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260927 13:43:02.299885 3795 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37017
I20260927 13:43:02.299921 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37017 every 8 connection(s)
I20260927 13:43:02.300974 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:02.302639 3931 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42195
I20260927 13:43:02.302843 3931 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:02.303437 3931 heartbeater.cc:511] Master 127.1.237.254:42195 requested a full tablet report, sending...
I20260927 13:43:02.304100 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3795
I20260927 13:43:02.304291 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:02.305123 1975 external_mini_cluster.cc:1428] Setting key 58c9a6ec094dd2adfffaa82ef33aaac3
I20260927 13:43:02.305589 3724 ts_manager.cc:194] Registered new tserver with Master: ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017)
I20260927 13:43:02.307371 3724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:33621
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:02.341768 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:02.641196 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:02.641503 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:02.641685 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:02.648459 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:02.648610 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:02.648751 3935 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:43:02.648831 3935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:02.648936 3935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:02.658010 3935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:02.659201 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:02.660408 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:02.672821 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:02.672967 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:02.672873 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:02.676328 3935 server_base.cc:1034] running on GCE node
I20260927 13:43:02.676864 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:02.677736 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:02.678938 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790516582678902 us; error 44 us; skew 500 ppm
I20260927 13:43:02.679275 3935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:02 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516582, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:02.682147 3935 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:02.683584 3935 webserver.cc:460] Webserver started at http://127.1.237.194:38781/ using document root <none> and password file <none>
I20260927 13:43:02.684288 3935 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:02.684422 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:02.684708 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:02.687422 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "5ee2b234d6784828986f9e4f3f2d2698"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "f84c4ea2eff12d9e93c941b64724e70e"
server_key_iv: "f14ee29e87d840b0a508d67ab8ddf28f"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.688123 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "5ee2b234d6784828986f9e4f3f2d2698"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "f84c4ea2eff12d9e93c941b64724e70e"
server_key_iv: "f14ee29e87d840b0a508d67ab8ddf28f"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.692737 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:02.695834 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:02.697010 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:02.697175 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "5ee2b234d6784828986f9e4f3f2d2698"
format_stamp: "Formatted at 2026-09-27 13:43:02 on dist-test-slave-7g3l"
server_key: "f84c4ea2eff12d9e93c941b64724e70e"
server_key_iv: "f14ee29e87d840b0a508d67ab8ddf28f"
server_key_version: "encryptionkey@0"
I20260927 13:43:02.697345 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:02.713276 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:02.717264 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:02.717624 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:02.720770 3935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:02.720902 3935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:02.721045 3935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:02.721182 3935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 13:43:02 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516582, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:02.744163 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.721210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:60201 (local address 127.1.237.254:42195)
0927 13:43:02.721405 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:02.721413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:02.722941 (+ 1528us) server_negotiation.cc:407] Connection header received
0927 13:43:02.725944 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:02.725950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:02.726024 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:02.726141 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:02.728295 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.729169 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.731818 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.732692 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.732850 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.737128 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:02.737137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:02.737145 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:02.737195 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:02.740957 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.741815 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.741820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.741823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.741943 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.742366 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.742371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.742374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.742789 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:02.742942 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:02.743649 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 13:43:02.743848 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:43:02.745112 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.721777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.194:60201)
0927 13:43:02.722750 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:02.722826 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:02.725581 (+ 2755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:02.726435 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:02.726449 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:02.727216 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:02.728065 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.728097 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.729340 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.729345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.731641 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.731656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.732931 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.732940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.733291 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.734136 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:02.734170 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:02.736885 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:02.741174 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.741185 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.741203 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.741688 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.742058 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.742063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.742066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.742231 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.742918 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:02.742926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:02.743307 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 13:43:02.743552 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 13:43:02.743997 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":374,"threads_started":1}
I20260927 13:43:02.766350 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.751810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.194:50517)
0927 13:43:02.752666 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:02.752685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:02.752795 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:02.753138 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:02.753141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:02.753583 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:02.753862 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.753870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.755553 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.755561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.757058 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:02.757080 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.758235 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.758240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:02.758356 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.758907 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:02.758923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:02.759827 (+ 904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:02.763377 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.763383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.763387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.763773 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.764291 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:02.764296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:02.764299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:02.764425 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:02.765246 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:02.765252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:02.765473 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 13:43:02.765661 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 13:43:02.766013 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":565,"threads_started":1}
I20260927 13:43:02.767081 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:02.751754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:50517 (local address 127.1.237.254:42195)
0927 13:43:02.751935 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:02.751943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:02.752776 (+ 833us) server_negotiation.cc:407] Connection header received
0927 13:43:02.752909 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:02.752917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:02.753005 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:02.753128 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:02.754023 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.755406 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.757197 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:02.758067 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:02.758216 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:02.760081 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:02.760086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:02.760089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:02.760114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:02.763263 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.763909 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.763929 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.763934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.764143 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:02.764619 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:02.764624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:02.764629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:02.764997 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:02.766622 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:02.766725 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 13:43:02.766874 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:43:02.776051 3935 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:37575
I20260927 13:43:02.776060 4069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:37575 every 8 connection(s)
I20260927 13:43:02.777177 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:02.778950 4070 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42195
I20260927 13:43:02.779208 4070 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:02.779806 4070 heartbeater.cc:511] Master 127.1.237.254:42195 requested a full tablet report, sending...
I20260927 13:43:02.781142 3724 ts_manager.cc:194] Registered new tserver with Master: 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:02.781821 3724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:60201
I20260927 13:43:02.785676 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 3935
I20260927 13:43:02.785799 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:02.786152 1975 external_mini_cluster.cc:1428] Setting key d2666488c5db07b4b9e36b9c6d0ecd24
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:02.824182 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:03.129710 4074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:03.129992 4074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:03.130163 4074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:03.137174 4074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:03.137295 4074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:03.137434 4074 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 13:43:03.137518 4074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:03.137639 4074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:03.146598 4074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:03.147770 4074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:03.148962 4074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:03.162087 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:03.162165 4080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:03.162169 4079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:03.162725 4074 server_base.cc:1034] running on GCE node
I20260927 13:43:03.163347 4074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:03.164160 4074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:03.165400 4074 hybrid_clock.cc:648] HybridClock initialized: now 1790516583165353 us; error 47 us; skew 500 ppm
I20260927 13:43:03.165727 4074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:03 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:03.168781 4074 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:03.170184 4074 webserver.cc:460] Webserver started at http://127.1.237.195:33787/ using document root <none> and password file <none>
I20260927 13:43:03.170872 4074 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:03.171013 4074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:03.171430 4074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:03.174427 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "ee9ab148fcee4fb88bab92c78bd05c5f"
format_stamp: "Formatted at 2026-09-27 13:43:03 on dist-test-slave-7g3l"
server_key: "ed4b23a22b8e545eee34123c39898b09"
server_key_iv: "7e3a4e08eea411d3023ddc12324e8d06"
server_key_version: "encryptionkey@0"
I20260927 13:43:03.175220 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "ee9ab148fcee4fb88bab92c78bd05c5f"
format_stamp: "Formatted at 2026-09-27 13:43:03 on dist-test-slave-7g3l"
server_key: "ed4b23a22b8e545eee34123c39898b09"
server_key_iv: "7e3a4e08eea411d3023ddc12324e8d06"
server_key_version: "encryptionkey@0"
I20260927 13:43:03.180377 4074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 13:43:03.183629 4089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:03.184757 4074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 13:43:03.184921 4074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "ee9ab148fcee4fb88bab92c78bd05c5f"
format_stamp: "Formatted at 2026-09-27 13:43:03 on dist-test-slave-7g3l"
server_key: "ed4b23a22b8e545eee34123c39898b09"
server_key_iv: "7e3a4e08eea411d3023ddc12324e8d06"
server_key_version: "encryptionkey@0"
I20260927 13:43:03.185112 4074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:03.220080 4074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:03.223843 4074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:03.224099 4074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:03.226419 4074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:03.226548 4074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:03.226671 4074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:03.226765 4074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:43:03 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:03.250078 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.226988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:56709 (local address 127.1.237.254:42195)
0927 13:43:03.227246 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.227255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.228747 (+ 1492us) server_negotiation.cc:407] Connection header received
0927 13:43:03.231684 (+ 2937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.231692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.231790 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.231897 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:03.234421 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.235319 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.237692 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.238220 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.238344 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.243060 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:03.243067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:03.243074 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:03.243116 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:03.246342 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.247034 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.247039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.247042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.247204 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.247752 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.247759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.247765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.248231 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:03.248420 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.249236 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.249554 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:43:03.250456 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.227756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.195:56709)
0927 13:43:03.228572 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.228622 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.231378 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.232236 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.232250 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.233235 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:03.234228 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.234254 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.235525 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.235529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.237553 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.237571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.238573 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.238578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.238738 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.239389 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:03.239423 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:03.242821 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:03.246491 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.246499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.246521 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.246904 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.247310 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.247314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.247318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.247610 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.248385 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:03.248390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:03.248770 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.249042 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.249378 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":307,"threads_started":1}
I20260927 13:43:03.270112 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.256291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.1.237.195:35121)
0927 13:43:03.256907 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.256944 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.257038 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.257357 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.257360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.257747 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:03.258001 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.258009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.259507 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.259524 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.260922 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.260937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.261647 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.261653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.261829 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.262423 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:03.262438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:03.263038 (+ 600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:03.266962 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.266965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.266967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.267382 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.268160 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.268167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.268172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.268359 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.269042 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:03.269047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:03.269258 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.269456 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.269794 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":363,"threads_started":1}
I20260927 13:43:03.270169 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.256175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:35121 (local address 127.1.237.254:42195)
0927 13:43:03.256344 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.256351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.257000 (+ 649us) server_negotiation.cc:407] Connection header received
0927 13:43:03.257137 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.257144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.257232 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.257350 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:03.258124 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.259369 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.261072 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.261471 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.261612 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.263244 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:03.263250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:03.263258 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:03.263285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:03.266829 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.267840 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.267849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.267855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.268010 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.268543 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.268550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.268556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.268922 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:03.269084 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.269581 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.269792 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:43:03.278337 4074 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:39781
I20260927 13:43:03.278368 4208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:39781 every 8 connection(s)
I20260927 13:43:03.279646 4074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:03.281376 4209 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42195
I20260927 13:43:03.281572 4209 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:03.282097 4209 heartbeater.cc:511] Master 127.1.237.254:42195 requested a full tablet report, sending...
I20260927 13:43:03.283274 3724 ts_manager.cc:194] Registered new tserver with Master: ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781)
I20260927 13:43:03.283968 3724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:56709
I20260927 13:43:03.288882 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4074
I20260927 13:43:03.289021 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:03.289485 1975 external_mini_cluster.cc:1428] Setting key c761098801a47e74c41e381613a3a123
I20260927 13:43:03.291985 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:43:03.310541 3931 heartbeater.cc:503] Master 127.1.237.254:42195 was elected leader, sending a full tablet report...
Sep 27 13:43:03 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:43:03 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:03.333813 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.321559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.1.237.254:42195)
0927 13:43:03.321806 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.321812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.322016 (+ 204us) server_negotiation.cc:407] Connection header received
0927 13:43:03.322146 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.322150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.322227 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.322303 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:03.323522 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.324212 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.325325 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.325748 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.325840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.329259 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:03.329263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:03.329266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:03.329294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:03.331577 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.332181 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.332184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.332186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.332299 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.332588 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.332591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.332593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.332812 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:03.332995 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.333306 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.333446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:43:03.337373 3724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:03.343804 3724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:03.364208 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.354823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.1.237.195:39781)
0927 13:43:03.355506 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.355515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.355558 (+ 43us) server_negotiation.cc:407] Connection header received
0927 13:43:03.355650 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.355673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.355911 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.356092 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.357314 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.358948 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.361813 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.363051 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.363188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.363396 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.363741 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.363885 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":352,"threads_started":1}
I20260927 13:43:03.364233 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.354486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39781 (local address 127.0.0.1:43608)
0927 13:43:03.355274 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.355320 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.355492 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.356580 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.356591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.356629 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.357077 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.357092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.359107 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.359113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.361624 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.361641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.363266 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.363272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.363521 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.363587 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.363693 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.363855 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":21,"thread_start_us":388,"threads_started":1}
I20260927 13:43:03.366021 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.355688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37017 (local address 127.0.0.1:53552)
0927 13:43:03.357722 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.357739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.357850 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.358605 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.358609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.358630 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.359010 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.359023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.360860 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.360867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.363501 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.363513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.365306 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.365312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.365475 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.365509 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.365642 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.365758 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903,"mutex_wait_us":1,"thread_start_us":1759,"threads_started":1}
I20260927 13:43:03.366194 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.355896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.1.237.193:37017)
0927 13:43:03.357476 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.357483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.357959 (+ 476us) server_negotiation.cc:407] Connection header received
0927 13:43:03.358076 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.358083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.358265 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.358420 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.359211 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.360718 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.363640 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.365095 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.365229 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.365408 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.365747 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.365910 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1220,"threads_started":1}
I20260927 13:43:03.368224 4139 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a (DEFAULT_TABLE table=test-table [id=223903805cc240dda205cdfae7004fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.369612 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec45bbd0a0140088a96cbce903cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.369413 3861 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a (DEFAULT_TABLE table=test-table [id=223903805cc240dda205cdfae7004fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.370724 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec45bbd0a0140088a96cbce903cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.370887 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.358033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.1.237.194:37575)
0927 13:43:03.361374 (+ 3341us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.361383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.361997 (+ 614us) server_negotiation.cc:407] Connection header received
0927 13:43:03.362087 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.362093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.362290 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.362452 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.363035 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.364716 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.368134 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.369753 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.369896 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.370102 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.370443 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.370606 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3183,"thread_start_us":3040,"threads_started":1}
I20260927 13:43:03.370903 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.357777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:37575 (local address 127.0.0.1:58072)
0927 13:43:03.361765 (+ 3988us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.361782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.361889 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.362454 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.362458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.362481 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.362861 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.362871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.364872 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.364881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.367948 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.367961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.369949 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.369955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.370246 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.370276 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.370367 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.370478 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3855,"thread_start_us":3721,"threads_started":1}
I20260927 13:43:03.374384 4000 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a (DEFAULT_TABLE table=test-table [id=223903805cc240dda205cdfae7004fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.375669 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec45bbd0a0140088a96cbce903cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.381021 4228 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Bootstrap starting.
I20260927 13:43:03.384094 4228 tablet_bootstrap.cc:654] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.385005 4228 log.cc:824] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:03.385818 4229 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Bootstrap starting.
I20260927 13:43:03.389379 4229 tablet_bootstrap.cc:654] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.389698 4228 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: No bootstrap required, opened a new log
I20260927 13:43:03.389791 4231 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Bootstrap starting.
I20260927 13:43:03.389972 4228 ts_tablet_manager.cc:1397] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:43:03.391206 4229 log.cc:824] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:03.392887 4231 tablet_bootstrap.cc:654] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.393769 4231 log.cc:824] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:03.394060 4229 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: No bootstrap required, opened a new log
I20260927 13:43:03.394312 4229 ts_tablet_manager.cc:1397] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:43:03.396067 4231 tablet_bootstrap.cc:492] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: No bootstrap required, opened a new log
I20260927 13:43:03.396297 4231 ts_tablet_manager.cc:1397] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 13:43:03.399917 4228 raft_consensus.cc:348] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.400329 4228 raft_consensus.cc:374] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.400416 4228 raft_consensus.cc:729] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.401000 4228 consensus_queue.cc:260] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.402072 4209 heartbeater.cc:503] Master 127.1.237.254:42195 was elected leader, sending a full tablet report...
I20260927 13:43:03.402601 4228 ts_tablet_manager.cc:1428] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 13:43:03.404628 4229 raft_consensus.cc:348] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.405005 4229 raft_consensus.cc:374] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.405089 4229 raft_consensus.cc:729] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac79b63a036a4074b1550dcbb846a115, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.405616 4229 consensus_queue.cc:260] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.405731 4231 raft_consensus.cc:348] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.406108 4231 raft_consensus.cc:374] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.406222 4231 raft_consensus.cc:729] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.406845 4231 consensus_queue.cc:260] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.407793 4070 heartbeater.cc:503] Master 127.1.237.254:42195 was elected leader, sending a full tablet report...
I20260927 13:43:03.408030 4229 ts_tablet_manager.cc:1428] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 13:43:03.408325 4231 ts_tablet_manager.cc:1428] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 13:43:03.521452 4234 raft_consensus.cc:482] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:03.521757 4234 raft_consensus.cc:504] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.523293 4234 leader_election.cc:288] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
W20260927 13:43:03.529093 4071 tablet.cc:2369] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:03.529775 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.523742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:60153 (local address 127.1.237.193:37017)
0927 13:43:03.524070 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.524075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.524095 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:43:03.524230 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.524236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.524307 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.524402 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.524938 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.525948 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.527775 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.528891 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.529043 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.529114 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.529424 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.529548 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260927 13:43:03.529775 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.523634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37017 (local address 127.1.237.195:60153)
0927 13:43:03.523920 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.523941 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.524094 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.524464 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.524472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.524519 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.524790 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.524800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.526079 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.526084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.527635 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.527645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.529029 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.529035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.529262 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.529290 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.529401 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.529530 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 13:43:03.530681 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.524187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:56329 (local address 127.1.237.194:37575)
0927 13:43:03.524381 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.524389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.524976 (+ 587us) server_negotiation.cc:407] Connection header received
0927 13:43:03.525130 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.525134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.525194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.525261 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.525759 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.526742 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.528710 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.529924 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.530080 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.530151 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.530435 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.530520 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:43:03.530879 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.524097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:37575 (local address 127.1.237.195:56329)
0927 13:43:03.524868 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.524886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.525005 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.525298 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.525302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.525324 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.525627 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.525637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.526863 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.526866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.528559 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.528573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.530146 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.530152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.530279 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.530301 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.530414 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.530539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":472,"threads_started":1}
W20260927 13:43:03.531013 4210 tablet.cc:2369] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:03.530829 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec45bbd0a0140088a96cbce903cea3a" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115" is_pre_election: true
I20260927 13:43:03.531428 3886 raft_consensus.cc:2454] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:03.532137 4092 leader_election.cc:302] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac79b63a036a4074b1550dcbb846a115, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:03.532032 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec45bbd0a0140088a96cbce903cea3a" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698" is_pre_election: true
I20260927 13:43:03.532500 4234 raft_consensus.cc:2781] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:03.532526 4025 raft_consensus.cc:2454] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:03.532645 4234 raft_consensus.cc:482] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:03.532744 4234 raft_consensus.cc:3037] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.534905 4234 raft_consensus.cc:504] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.535470 4234 leader_election.cc:288] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Requested vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:03.535871 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec45bbd0a0140088a96cbce903cea3a" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115"
I20260927 13:43:03.535936 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec45bbd0a0140088a96cbce903cea3a" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698"
I20260927 13:43:03.536096 4025 raft_consensus.cc:3037] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.536100 3886 raft_consensus.cc:3037] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.538100 4025 raft_consensus.cc:2454] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:03.538193 3886 raft_consensus.cc:2454] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:03.538507 4092 leader_election.cc:302] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:03.538753 4234 raft_consensus.cc:2781] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:03.539208 4234 raft_consensus.cc:686] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Becoming Leader. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Running, Role: LEADER
I20260927 13:43:03.539685 4234 consensus_queue.cc:237] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } }
I20260927 13:43:03.543905 3724 catalog_manager.cc:5495] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195). New cstate: current_term: 1 leader_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } health_report { overall_health: UNKNOWN } } }
W20260927 13:43:03.553083 3932 tablet.cc:2369] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:43:03 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:03.585284 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.573384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42195 (local address 127.0.0.1:43172)
0927 13:43:03.573891 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.573904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.574057 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.574471 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.574474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.575087 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:03.575371 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.575379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.576374 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.576377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.577318 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.577329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.577965 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.577969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.578088 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.578699 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:03.578715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:03.580774 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:03.583312 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.583320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.583326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.583712 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.584012 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:03.584015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:03.584017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:03.584127 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:03.584532 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:03.584537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:03.584662 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.584780 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.584948 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260927 13:43:03.585290 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.573362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.1.237.254:42195)
0927 13:43:03.573518 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.573522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.573992 (+ 470us) server_negotiation.cc:407] Connection header received
0927 13:43:03.574198 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.574205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.574278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.574330 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:03.575512 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.576266 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.577419 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.577818 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.577905 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.580925 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:03.580929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:03.580932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:03.580958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:03.583057 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.583797 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.583800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.583803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.583930 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:03.584210 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:03.584213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:03.584215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:03.584435 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:03.584529 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.584868 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.585039 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":52}
I20260927 13:43:03.589699 3724 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43172:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:43:03.590643 3724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:03.596347 3861 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e7117cb145aab1cb640457c968f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43febb9a03b4d988016588749cf850b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:03.596534 4000 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e7117cb145aab1cb640457c968f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43febb9a03b4d988016588749cf850b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:03.597193 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e7117cb145aab1cb640457c968f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.596896 4139 tablet_service.cc:1467] Processing CreateTablet for tablet a9e9e7117cb145aab1cb640457c968f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c43febb9a03b4d988016588749cf850b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:03.597415 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e7117cb145aab1cb640457c968f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.597636 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e9e7117cb145aab1cb640457c968f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.602416 4229 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: Bootstrap starting.
I20260927 13:43:03.603720 4229 tablet_bootstrap.cc:654] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.604308 4231 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: Bootstrap starting.
I20260927 13:43:03.604753 4228 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: Bootstrap starting.
I20260927 13:43:03.605825 4231 tablet_bootstrap.cc:654] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.606278 4228 tablet_bootstrap.cc:654] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.611850 4228 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: No bootstrap required, opened a new log
I20260927 13:43:03.612004 4228 ts_tablet_manager.cc:1397] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260927 13:43:03.612064 4229 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: No bootstrap required, opened a new log
I20260927 13:43:03.612193 4229 ts_tablet_manager.cc:1397] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260927 13:43:03.612517 4231 tablet_bootstrap.cc:492] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: No bootstrap required, opened a new log
I20260927 13:43:03.612640 4231 ts_tablet_manager.cc:1397] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 13:43:03.612756 4228 raft_consensus.cc:348] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.612960 4228 raft_consensus.cc:374] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.613047 4228 raft_consensus.cc:729] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.612946 4229 raft_consensus.cc:348] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.613159 4229 raft_consensus.cc:374] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.613243 4229 raft_consensus.cc:729] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac79b63a036a4074b1550dcbb846a115, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.613258 4228 consensus_queue.cc:260] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.613355 4231 raft_consensus.cc:348] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.613574 4231 raft_consensus.cc:374] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.613487 4229 consensus_queue.cc:260] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.613651 4231 raft_consensus.cc:729] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.613826 4228 ts_tablet_manager.cc:1428] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:03.613835 4231 consensus_queue.cc:260] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.613902 4235 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.614219 4235 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.614130 4234 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.614364 4234 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.614271 4236 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.614585 4236 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.615258 4229 ts_tablet_manager.cc:1428] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 13:43:03.615654 4231 ts_tablet_manager.cc:1428] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:43:03.626600 4236 raft_consensus.cc:482] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:03.626762 4236 raft_consensus.cc:504] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.628332 4236 leader_election.cc:288] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781)
I20260927 13:43:03.635952 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.628676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:50527 (local address 127.1.237.193:37017)
0927 13:43:03.628825 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.628829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.629181 (+ 352us) server_negotiation.cc:407] Connection header received
0927 13:43:03.629494 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.629500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.629567 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.629629 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.630214 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.631264 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.633844 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.635022 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.635224 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.635296 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.635596 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.635686 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:43:03.636124 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.629349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:47749 (local address 127.1.237.195:39781)
0927 13:43:03.629545 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.629552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.629790 (+ 238us) server_negotiation.cc:407] Connection header received
0927 13:43:03.629968 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.629973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.630033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.630120 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:03.630621 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.631727 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.634051 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.635286 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.635415 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.635458 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.635773 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.635895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:43:03.636179 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.628507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37017 (local address 127.1.237.194:50527)
0927 13:43:03.629095 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.629114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.629283 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.629775 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.629778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.629795 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.630069 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.630076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.631399 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.631405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.633715 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.633729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.635257 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.635261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.635418 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.635462 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.635547 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.635669 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":3,"thread_start_us":300,"threads_started":1}
I20260927 13:43:03.636560 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.629179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39781 (local address 127.1.237.194:47749)
0927 13:43:03.629707 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.629719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.629850 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.630132 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.630136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.630157 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:03.630442 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.630449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.631866 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.631873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.633925 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.633934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.635462 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.635467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.635632 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.635661 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.635736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.635911 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":260,"threads_started":1}
I20260927 13:43:03.636765 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e7117cb145aab1cb640457c968f7" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115" is_pre_election: true
I20260927 13:43:03.636974 3886 raft_consensus.cc:2454] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 0.
I20260927 13:43:03.637303 4164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e7117cb145aab1cb640457c968f7" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" is_pre_election: true
I20260927 13:43:03.637387 3953 leader_election.cc:302] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ac79b63a036a4074b1550dcbb846a115; no voters:
I20260927 13:43:03.637648 4164 raft_consensus.cc:2454] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 0.
I20260927 13:43:03.637732 4236 raft_consensus.cc:2781] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:03.637832 4236 raft_consensus.cc:482] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:03.637929 4236 raft_consensus.cc:3037] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.639916 4236 raft_consensus.cc:504] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.640381 4236 leader_election.cc:288] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 election: Requested vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781)
I20260927 13:43:03.640718 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e7117cb145aab1cb640457c968f7" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115"
I20260927 13:43:03.640802 4164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e9e7117cb145aab1cb640457c968f7" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f"
I20260927 13:43:03.640913 3886 raft_consensus.cc:3037] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.640982 4164 raft_consensus.cc:3037] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:03.642899 3886 raft_consensus.cc:2454] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 1.
I20260927 13:43:03.643077 4164 raft_consensus.cc:2454] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 1.
I20260927 13:43:03.643407 3953 leader_election.cc:302] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ac79b63a036a4074b1550dcbb846a115; no voters:
I20260927 13:43:03.643658 4236 raft_consensus.cc:2781] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:03.644040 4236 raft_consensus.cc:686] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Running, Role: LEADER
I20260927 13:43:03.644454 4236 consensus_queue.cc:237] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.647425 4251 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: TxnStatusTablet state changed. Reason: New leader 5ee2b234d6784828986f9e4f3f2d2698. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.647792 4251 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:03.648465 4253 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:43:03.648715 4253 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:43:03.648487 3722 catalog_manager.cc:5495] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:03.665592 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.660637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.1.237.194:37575)
0927 13:43:03.660780 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.660786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.660827 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:43:03.661092 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.661097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.661177 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.661300 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:03.661617 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.662343 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.663640 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.664020 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.664176 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.664567 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.664983 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.665153 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.665213 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.665283 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:03.665614 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.660562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:37575 (local address 127.0.0.1:58280)
0927 13:43:03.660755 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.660775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.660917 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.661315 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.661319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.661330 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:03.661502 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.661510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.662454 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.662462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.663505 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.663515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.664185 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.664190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.664359 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.664405 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.665107 (+ 702us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.665125 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.665195 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.665324 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 13:43:03.673219 4164 raft_consensus.cc:1261] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Refusing update from remote peer 5ee2b234d6784828986f9e4f3f2d2698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:03.673365 3886 raft_consensus.cc:1261] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Refusing update from remote peer 5ee2b234d6784828986f9e4f3f2d2698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:03.674007 4251 consensus_queue.cc:1035] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:03.674405 4236 consensus_queue.cc:1035] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:03.685032 4235 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: TxnStatusTablet state changed. Reason: New leader 5ee2b234d6784828986f9e4f3f2d2698. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.685277 4235 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.689579 4236 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.689836 4236 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:03.696659 4235 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.696871 4235 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ac79b63a036a4074b1550dcbb846a115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.699885 4234 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: TxnStatusTablet state changed. Reason: New leader 5ee2b234d6784828986f9e4f3f2d2698. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.700174 4234 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.701938 4251 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.702188 4251 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P 5ee2b234d6784828986f9e4f3f2d2698: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:03.707682 4234 tablet_replica.cc:439] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } } }
I20260927 13:43:03.707954 4234 tablet_replica.cc:442] T a9e9e7117cb145aab1cb640457c968f7 P ee9ab148fcee4fb88bab92c78bd05c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:03.709740 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.701119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.1.237.195:39781)
0927 13:43:03.701296 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.701304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.701326 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:43:03.701404 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.701409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.701512 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.701656 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:03.702454 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.703689 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.705778 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.706397 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.706505 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.707063 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.707682 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.707812 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.709379 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.709457 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:43:03.727876 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.723114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:46381 (local address 127.1.237.194:37575)
0927 13:43:03.723326 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.723331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.723522 (+ 191us) server_negotiation.cc:407] Connection header received
0927 13:43:03.723722 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.723727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.723803 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.723896 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:03.724227 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.724947 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.726223 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.726645 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.726761 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.727210 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.727309 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.727401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.727508 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.727582 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:43:03.727926 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.723259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:37575 (local address 127.1.237.195:46381)
0927 13:43:03.723434 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.723449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.723587 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.723916 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.723920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.723931 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:03.724100 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.724108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.725081 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.725088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.726083 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.726093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.726789 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.726794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.726930 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.726975 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.727401 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.727415 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.727490 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.727635 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 13:43:03.740607 4025 raft_consensus.cc:1261] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:03.741919 4234 consensus_queue.cc:1035] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:03.745325 3886 raft_consensus.cc:1261] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:03.745888 4234 consensus_queue.cc:1035] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:03.747521 4258 mvcc.cc:205] Tried to move back new op lower bound from 7333955926991384576 to 7333955926182404096. Current Snapshot: MvccSnapshot[applied={T|T < 7333955926991384576}]
I20260927 13:43:03.761849 4257 mvcc.cc:205] Tried to move back new op lower bound from 7333955926991384576 to 7333955926182404096. Current Snapshot: MvccSnapshot[applied={T|T < 7333955926991384576}]
I20260927 13:43:03.797111 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.791208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:59321 (local address 127.1.237.195:39781)
0927 13:43:03.791397 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:03.791405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:03.791862 (+ 457us) server_negotiation.cc:407] Connection header received
0927 13:43:03.792082 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:03.792087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:03.792160 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:03.792244 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:03.792719 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.793379 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.795290 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.795787 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.795918 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.796452 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.796556 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.796643 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:03.796749 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 13:43:03.796830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:43:03.797273 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:03.791223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39781 (local address 127.1.237.194:59321)
0927 13:43:03.791772 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:03.791787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:03.791937 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:03.792256 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:03.792260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:03.792271 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:03.792580 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.792588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.793492 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.793499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.795128 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:03.795141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:03.795964 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:03.795977 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:03.796201 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:03.796275 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:03.796644 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:03.796657 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 13:43:03.796732 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:43:03.796858 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260927 13:43:03.835361 3724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:03.835634 3724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:03.839319 3724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Sending DeleteTablet for 3 replicas of tablet 3ec45bbd0a0140088a96cbce903cea3a
I20260927 13:43:03.840519 4139 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:03 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43608
I20260927 13:43:03.840698 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:03 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260927 13:43:03.842568 4000 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec45bbd0a0140088a96cbce903cea3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:03 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58072
I20260927 13:43:03.847110 4277 tablet_replica.cc:330] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: stopping tablet replica
I20260927 13:43:03.847002 3724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:03.847657 3724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:03.847599 4277 raft_consensus.cc:2229] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:03.848634 4277 raft_consensus.cc:2258] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:03.851475 4278 tablet_replica.cc:330] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: stopping tablet replica
I20260927 13:43:03.852463 4278 raft_consensus.cc:2229] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:03.853145 4277 ts_tablet_manager.cc:1905] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:03.853564 4278 raft_consensus.cc:2258] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:03.854740 4138 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6cda9a608452eb64346989d53f14f (DEFAULT_TABLE table=test-table [id=7e93faf12abb4905a96e4b88411dba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.854951 3861 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6cda9a608452eb64346989d53f14f (DEFAULT_TABLE table=test-table [id=7e93faf12abb4905a96e4b88411dba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.855304 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6cda9a608452eb64346989d53f14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.855365 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6cda9a608452eb64346989d53f14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.856731 4278 ts_tablet_manager.cc:1905] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:03.857530 4276 tablet_replica.cc:330] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: stopping tablet replica
I20260927 13:43:03.859947 3999 tablet_service.cc:1467] Processing CreateTablet for tablet 49d6cda9a608452eb64346989d53f14f (DEFAULT_TABLE table=test-table [id=7e93faf12abb4905a96e4b88411dba00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:03.860374 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d6cda9a608452eb64346989d53f14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:03.860517 4276 raft_consensus.cc:2229] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:03.861199 4276 raft_consensus.cc:2258] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:03.861862 4228 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Bootstrap starting.
I20260927 13:43:03.863315 4276 ts_tablet_manager.cc:1905] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:03.863497 4228 tablet_bootstrap.cc:654] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.870935 4231 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Bootstrap starting.
I20260927 13:43:03.872435 4277 ts_tablet_manager.cc:1918] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:03.872597 4277 log.cc:1196] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/3ec45bbd0a0140088a96cbce903cea3a
I20260927 13:43:03.873099 4277 ts_tablet_manager.cc:1939] T 3ec45bbd0a0140088a96cbce903cea3a P ac79b63a036a4074b1550dcbb846a115: Deleting consensus metadata
I20260927 13:43:03.873229 4231 tablet_bootstrap.cc:654] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.873998 4278 ts_tablet_manager.cc:1918] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:03.874197 4278 log.cc:1196] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/3ec45bbd0a0140088a96cbce903cea3a
I20260927 13:43:03.874935 3710 catalog_manager.cc:4841] TS ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017): tablet 3ec45bbd0a0140088a96cbce903cea3a (table test-table [id=223903805cc240dda205cdfae7004fda]) successfully deleted
I20260927 13:43:03.875384 4278 ts_tablet_manager.cc:1939] T 3ec45bbd0a0140088a96cbce903cea3a P 5ee2b234d6784828986f9e4f3f2d2698: Deleting consensus metadata
I20260927 13:43:03.878319 4228 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: No bootstrap required, opened a new log
I20260927 13:43:03.878507 4228 ts_tablet_manager.cc:1397] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20260927 13:43:03.879002 4276 ts_tablet_manager.cc:1918] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:03.879037 3710 catalog_manager.cc:4841] TS 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575): tablet 3ec45bbd0a0140088a96cbce903cea3a (table test-table [id=223903805cc240dda205cdfae7004fda]) successfully deleted
I20260927 13:43:03.879268 4276 log.cc:1196] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/3ec45bbd0a0140088a96cbce903cea3a
I20260927 13:43:03.879441 4228 raft_consensus.cc:348] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.879711 4228 raft_consensus.cc:374] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.879798 4228 raft_consensus.cc:729] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.880105 4276 ts_tablet_manager.cc:1939] T 3ec45bbd0a0140088a96cbce903cea3a P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting consensus metadata
I20260927 13:43:03.880074 4228 consensus_queue.cc:260] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.880686 4228 ts_tablet_manager.cc:1428] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 13:43:03.882810 4229 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Bootstrap starting.
I20260927 13:43:03.884318 4229 tablet_bootstrap.cc:654] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:03.886332 3710 catalog_manager.cc:4841] TS ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781): tablet 3ec45bbd0a0140088a96cbce903cea3a (table test-table [id=223903805cc240dda205cdfae7004fda]) successfully deleted
I20260927 13:43:03.887778 4229 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: No bootstrap required, opened a new log
I20260927 13:43:03.887969 4229 ts_tablet_manager.cc:1397] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 13:43:03.888752 4229 raft_consensus.cc:348] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.889066 4229 raft_consensus.cc:374] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.889185 4229 raft_consensus.cc:729] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac79b63a036a4074b1550dcbb846a115, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.889528 4229 consensus_queue.cc:260] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.890864 4229 ts_tablet_manager.cc:1428] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 13:43:03.891602 4231 tablet_bootstrap.cc:492] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: No bootstrap required, opened a new log
I20260927 13:43:03.891973 4231 ts_tablet_manager.cc:1397] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20260927 13:43:03.892709 4231 raft_consensus.cc:348] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.893103 4231 raft_consensus.cc:374] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:03.893196 4231 raft_consensus.cc:729] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Initialized, Role: FOLLOWER
I20260927 13:43:03.893440 4231 consensus_queue.cc:260] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:03.894039 4231 ts_tablet_manager.cc:1428] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:04.124727 4267 raft_consensus.cc:482] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:04.124950 4267 raft_consensus.cc:504] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:04.125761 4267 leader_election.cc:288] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781)
I20260927 13:43:04.126243 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6cda9a608452eb64346989d53f14f" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115" is_pre_election: true
I20260927 13:43:04.126294 4164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6cda9a608452eb64346989d53f14f" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" is_pre_election: true
I20260927 13:43:04.126490 3886 raft_consensus.cc:2454] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 0.
I20260927 13:43:04.126533 4164 raft_consensus.cc:2454] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 0.
I20260927 13:43:04.126916 3953 leader_election.cc:302] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ac79b63a036a4074b1550dcbb846a115; no voters:
I20260927 13:43:04.127184 4267 raft_consensus.cc:2781] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:04.127302 4267 raft_consensus.cc:482] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:04.127413 4267 raft_consensus.cc:3037] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:04.129281 4267 raft_consensus.cc:504] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:04.129782 4267 leader_election.cc:288] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 election: Requested vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781)
I20260927 13:43:04.130179 3886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6cda9a608452eb64346989d53f14f" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115"
I20260927 13:43:04.130375 3886 raft_consensus.cc:3037] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:04.130339 4164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d6cda9a608452eb64346989d53f14f" candidate_uuid: "5ee2b234d6784828986f9e4f3f2d2698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f"
I20260927 13:43:04.130515 4164 raft_consensus.cc:3037] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:04.132532 3886 raft_consensus.cc:2454] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 1.
I20260927 13:43:04.132898 3953 leader_election.cc:302] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ac79b63a036a4074b1550dcbb846a115; no voters:
I20260927 13:43:04.133188 4267 raft_consensus.cc:2781] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:04.133338 4267 raft_consensus.cc:686] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Running, Role: LEADER
I20260927 13:43:04.133842 4267 consensus_queue.cc:237] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:04.133987 4164 raft_consensus.cc:2454] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2b234d6784828986f9e4f3f2d2698 in term 1.
I20260927 13:43:04.136651 3724 catalog_manager.cc:5495] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "5ee2b234d6784828986f9e4f3f2d2698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:04.182644 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:04.177885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.1.237.194:37575)
0927 13:43:04.178021 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:04.178026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:04.178296 (+ 270us) server_negotiation.cc:407] Connection header received
0927 13:43:04.178480 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:04.178484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:04.178562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:04.178649 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:04.179042 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.179818 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.181056 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.181442 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.181579 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:04.181951 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:04.182064 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:04.182146 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:04.182285 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 13:43:04.182352 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 13:43:04.195266 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:04.190374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:37575 (local address 127.1.237.194:43181)
0927 13:43:04.190589 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:04.190608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:04.190691 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:04.190975 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:04.190978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:04.190986 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:04.191233 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:04.191241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.192540 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.192543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:04.193482 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:04.193491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.194334 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.194338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:04.194442 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:04.194471 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:04.194835 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:04.194841 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:43:04.194895 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 13:43:04.194983 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":23}
I20260927 13:43:04.195302 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:04.190552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43181 (local address 127.1.237.194:37575)
0927 13:43:04.190711 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:04.190717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:04.190736 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:43:04.190790 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:04.190797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:04.190876 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:04.190969 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:04.191358 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.192420 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.193620 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:04.194210 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:04.194327 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:04.194619 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:04.194751 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:04.194836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:04.194916 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 13:43:04.194984 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":47}
I20260927 13:43:04.202747 4164 raft_consensus.cc:1261] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Refusing update from remote peer 5ee2b234d6784828986f9e4f3f2d2698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:04.203011 3886 raft_consensus.cc:1261] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Refusing update from remote peer 5ee2b234d6784828986f9e4f3f2d2698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:04.203333 4267 consensus_queue.cc:1035] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:04.203949 4267 consensus_queue.cc:1035] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:04.207078 4256 mvcc.cc:205] Tried to move back new op lower bound from 7333955928890007552 to 7333955928614481920. Current Snapshot: MvccSnapshot[applied={T|T < 7333955928890007552}]
W20260927 13:43:04.224364 4256 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:43:05.818217 3724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:05.818401 3724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:05.822139 3724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Sending DeleteTablet for 3 replicas of tablet 49d6cda9a608452eb64346989d53f14f
I20260927 13:43:05.822777 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d6cda9a608452eb64346989d53f14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260927 13:43:05.822932 3999 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d6cda9a608452eb64346989d53f14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58072
I20260927 13:43:05.822932 4138 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d6cda9a608452eb64346989d53f14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43608
I20260927 13:43:05.835271 4307 tablet_replica.cc:330] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: stopping tablet replica
I20260927 13:43:05.835626 4307 raft_consensus.cc:2229] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:05.835938 4307 raft_consensus.cc:2258] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:05.836963 4307 ts_tablet_manager.cc:1905] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:05.838510 4309 tablet_replica.cc:330] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: stopping tablet replica
I20260927 13:43:05.839010 4308 tablet_replica.cc:330] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: stopping tablet replica
I20260927 13:43:05.839257 4308 raft_consensus.cc:2229] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:05.839340 4309 raft_consensus.cc:2229] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:05.839603 4308 raft_consensus.cc:2258] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:05.839965 4309 raft_consensus.cc:2258] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:05.840642 4308 ts_tablet_manager.cc:1905] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:05.840896 4309 ts_tablet_manager.cc:1905] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:05.844240 4307 ts_tablet_manager.cc:1918] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:05.844425 4307 log.cc:1196] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/49d6cda9a608452eb64346989d53f14f
I20260927 13:43:05.844938 4307 ts_tablet_manager.cc:1939] T 49d6cda9a608452eb64346989d53f14f P ac79b63a036a4074b1550dcbb846a115: Deleting consensus metadata
I20260927 13:43:05.846119 3710 catalog_manager.cc:4841] TS ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017): tablet 49d6cda9a608452eb64346989d53f14f (table test-table [id=7e93faf12abb4905a96e4b88411dba00]) successfully deleted
I20260927 13:43:05.847113 4308 ts_tablet_manager.cc:1918] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:05.847101 4309 ts_tablet_manager.cc:1918] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:05.847252 4308 log.cc:1196] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/49d6cda9a608452eb64346989d53f14f
I20260927 13:43:05.847261 4309 log.cc:1196] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/49d6cda9a608452eb64346989d53f14f
I20260927 13:43:05.847698 4308 ts_tablet_manager.cc:1939] T 49d6cda9a608452eb64346989d53f14f P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting consensus metadata
I20260927 13:43:05.847728 4309 ts_tablet_manager.cc:1939] T 49d6cda9a608452eb64346989d53f14f P 5ee2b234d6784828986f9e4f3f2d2698: Deleting consensus metadata
I20260927 13:43:05.848694 3710 catalog_manager.cc:4841] TS 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575): tablet 49d6cda9a608452eb64346989d53f14f (table test-table [id=7e93faf12abb4905a96e4b88411dba00]) successfully deleted
I20260927 13:43:05.848927 3710 catalog_manager.cc:4841] TS ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781): tablet 49d6cda9a608452eb64346989d53f14f (table test-table [id=7e93faf12abb4905a96e4b88411dba00]) successfully deleted
Sep 27 13:43:05 dist-test-slave-7g3l krb5kdc[3679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:43:05 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:05.884203 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:05.865084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.1.237.254:42195)
0927 13:43:05.865570 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:05.865575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:05.865965 (+ 390us) server_negotiation.cc:407] Connection header received
0927 13:43:05.866105 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:05.866108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:05.866164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:05.866248 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:05.867553 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:05.868299 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:05.869431 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:05.869801 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:05.869888 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:05.879618 (+ 9730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:05.879623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:05.879626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:05.879656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:05.882106 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:05.882636 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:05.882640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:05.882642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:05.882753 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:05.883020 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:05.883023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:05.883025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:05.883327 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:05.883493 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:05.883721 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 13:43:05.883862 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260927 13:43:05.887365 3724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:05.887923 3724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:05.893678 3861 tablet_service.cc:1467] Processing CreateTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (DEFAULT_TABLE table=test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:05.893764 3999 tablet_service.cc:1467] Processing CreateTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (DEFAULT_TABLE table=test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:05.893956 4138 tablet_service.cc:1467] Processing CreateTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (DEFAULT_TABLE table=test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:05.894068 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:05.894181 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:05.894353 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:05.898468 4318 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Bootstrap starting.
I20260927 13:43:05.900002 4318 tablet_bootstrap.cc:654] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:05.900946 4319 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Bootstrap starting.
I20260927 13:43:05.901696 4320 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Bootstrap starting.
I20260927 13:43:05.903409 4318 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: No bootstrap required, opened a new log
I20260927 13:43:05.903546 4318 ts_tablet_manager.cc:1397] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 13:43:05.903697 4319 tablet_bootstrap.cc:654] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:05.903941 4320 tablet_bootstrap.cc:654] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:05.904178 4318 raft_consensus.cc:348] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.904356 4318 raft_consensus.cc:374] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:05.904455 4318 raft_consensus.cc:729] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Initialized, Role: FOLLOWER
I20260927 13:43:05.904734 4318 consensus_queue.cc:260] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.905329 4318 ts_tablet_manager.cc:1428] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:05.911612 4319 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: No bootstrap required, opened a new log
I20260927 13:43:05.911752 4319 ts_tablet_manager.cc:1397] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 13:43:05.911779 4320 tablet_bootstrap.cc:492] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: No bootstrap required, opened a new log
I20260927 13:43:05.911903 4320 ts_tablet_manager.cc:1397] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 13:43:05.912346 4319 raft_consensus.cc:348] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.912534 4319 raft_consensus.cc:374] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:05.912487 4320 raft_consensus.cc:348] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.912624 4319 raft_consensus.cc:729] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac79b63a036a4074b1550dcbb846a115, State: Initialized, Role: FOLLOWER
I20260927 13:43:05.912717 4320 raft_consensus.cc:374] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:05.912868 4320 raft_consensus.cc:729] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Initialized, Role: FOLLOWER
I20260927 13:43:05.912880 4319 consensus_queue.cc:260] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.913136 4320 consensus_queue.cc:260] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.914340 4320 ts_tablet_manager.cc:1428] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:43:05.914707 4319 ts_tablet_manager.cc:1428] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 13:43:05.916334 4325 raft_consensus.cc:482] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:05.916534 4325 raft_consensus.cc:504] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.917289 4325 leader_election.cc:288] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:05.917737 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b6d7dd3e8b47dd8ecef00e7c32e446" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115" is_pre_election: true
I20260927 13:43:05.917950 3885 raft_consensus.cc:2454] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:05.918248 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b6d7dd3e8b47dd8ecef00e7c32e446" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698" is_pre_election: true
I20260927 13:43:05.918476 4025 raft_consensus.cc:2454] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:05.918916 4092 leader_election.cc:302] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac79b63a036a4074b1550dcbb846a115, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:05.919222 4325 raft_consensus.cc:2781] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:05.919348 4325 raft_consensus.cc:482] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:05.919453 4325 raft_consensus.cc:3037] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:05.921183 4325 raft_consensus.cc:504] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.921655 4325 leader_election.cc:288] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Requested vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:05.922026 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b6d7dd3e8b47dd8ecef00e7c32e446" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115"
I20260927 13:43:05.922224 3885 raft_consensus.cc:3037] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:05.922175 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b6d7dd3e8b47dd8ecef00e7c32e446" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698"
I20260927 13:43:05.922353 4025 raft_consensus.cc:3037] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:05.925004 3885 raft_consensus.cc:2454] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:05.925104 4025 raft_consensus.cc:2454] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:05.925369 4092 leader_election.cc:302] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac79b63a036a4074b1550dcbb846a115, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:05.925603 4325 raft_consensus.cc:2781] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:05.925951 4325 raft_consensus.cc:686] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Becoming Leader. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Running, Role: LEADER
I20260927 13:43:05.926208 4325 consensus_queue.cc:237] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:05.928301 3724 catalog_manager.cc:5495] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195). New cstate: current_term: 1 leader_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } health_report { overall_health: HEALTHY } } }
I20260927 13:43:05.946256 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:05.941457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.1.237.195:39781)
0927 13:43:05.942393 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:05.942396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:05.942413 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:05.942456 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:05.942459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:05.942510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:05.942588 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:05.942870 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:05.943680 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:05.944879 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:05.945274 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:05.945359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:05.945699 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:05.945782 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:05.945843 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:05.945960 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 13:43:05.946011 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":724,"threads_started":1}
I20260927 13:43:05.954370 3885 raft_consensus.cc:1261] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:05.954370 4025 raft_consensus.cc:1261] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:05.954937 4326 consensus_queue.cc:1035] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:05.955232 4325 consensus_queue.cc:1035] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:05.996847 3722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:05.997081 3722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:05.999643 3722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Sending DeleteTablet for 3 replicas of tablet 03b6d7dd3e8b47dd8ecef00e7c32e446
I20260927 13:43:06.000305 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260927 13:43:06.000381 3999 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58072
I20260927 13:43:06.000437 4138 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:05 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43608
I20260927 13:43:06.000559 4307 tablet_replica.cc:330] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: stopping tablet replica
I20260927 13:43:06.000681 4308 tablet_replica.cc:330] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: stopping tablet replica
I20260927 13:43:06.000749 4307 raft_consensus.cc:2229] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:06.000867 4308 raft_consensus.cc:2229] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:06.000927 4307 raft_consensus.cc:2258] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:06.001173 4308 raft_consensus.cc:2258] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:06.001329 4309 tablet_replica.cc:330] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: stopping tablet replica
I20260927 13:43:06.001549 4309 raft_consensus.cc:2229] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:06.001754 4309 raft_consensus.cc:2258] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:06.001998 4307 ts_tablet_manager.cc:1905] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:06.001839 3722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 13:43:06.002161 4308 ts_tablet_manager.cc:1905] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 13:43:06.002637 3722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:06.003072 4309 ts_tablet_manager.cc:1905] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:06.008352 4308 ts_tablet_manager.cc:1918] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:06.008496 4308 log.cc:1196] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/03b6d7dd3e8b47dd8ecef00e7c32e446
I20260927 13:43:06.009346 4307 ts_tablet_manager.cc:1918] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:06.009581 4307 log.cc:1196] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/03b6d7dd3e8b47dd8ecef00e7c32e446
I20260927 13:43:06.009758 3861 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a5193fd1b4464a10effe027e0cebc (DEFAULT_TABLE table=test-table [id=0cb2f484c535474b94f8f32a851e7436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:06.009768 4138 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a5193fd1b4464a10effe027e0cebc (DEFAULT_TABLE table=test-table [id=0cb2f484c535474b94f8f32a851e7436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:06.010125 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a5193fd1b4464a10effe027e0cebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:06.010149 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a5193fd1b4464a10effe027e0cebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:06.010241 4307 ts_tablet_manager.cc:1939] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ac79b63a036a4074b1550dcbb846a115: Deleting consensus metadata
I20260927 13:43:06.010277 4309 ts_tablet_manager.cc:1918] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:06.010457 4309 log.cc:1196] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/03b6d7dd3e8b47dd8ecef00e7c32e446
I20260927 13:43:06.010825 4309 ts_tablet_manager.cc:1939] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P 5ee2b234d6784828986f9e4f3f2d2698: Deleting consensus metadata
I20260927 13:43:06.011487 3710 catalog_manager.cc:4841] TS ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017): tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (table test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]) successfully deleted
I20260927 13:43:06.012148 3710 catalog_manager.cc:4841] TS 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575): tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (table test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]) successfully deleted
I20260927 13:43:06.012341 3999 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a5193fd1b4464a10effe027e0cebc (DEFAULT_TABLE table=test-table [id=0cb2f484c535474b94f8f32a851e7436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:06.012689 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a5193fd1b4464a10effe027e0cebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:06.013325 4308 ts_tablet_manager.cc:1939] T 03b6d7dd3e8b47dd8ecef00e7c32e446 P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting consensus metadata
I20260927 13:43:06.014278 3710 catalog_manager.cc:4841] TS ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781): tablet 03b6d7dd3e8b47dd8ecef00e7c32e446 (table test-table [id=9c2b45265e614f0d870ce3eb92b9fa4e]) successfully deleted
I20260927 13:43:06.016554 4320 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Bootstrap starting.
I20260927 13:43:06.017393 4319 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Bootstrap starting.
I20260927 13:43:06.017706 4318 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Bootstrap starting.
I20260927 13:43:06.018095 4320 tablet_bootstrap.cc:654] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:06.018841 4319 tablet_bootstrap.cc:654] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:06.019141 4318 tablet_bootstrap.cc:654] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:06.020170 4320 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: No bootstrap required, opened a new log
I20260927 13:43:06.020318 4320 ts_tablet_manager.cc:1397] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 13:43:06.020730 4319 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: No bootstrap required, opened a new log
I20260927 13:43:06.020896 4319 ts_tablet_manager.cc:1397] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 13:43:06.020876 4320 raft_consensus.cc:348] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.021121 4320 raft_consensus.cc:374] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:06.021194 4320 raft_consensus.cc:729] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Initialized, Role: FOLLOWER
I20260927 13:43:06.021406 4320 consensus_queue.cc:260] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.021844 4320 ts_tablet_manager.cc:1428] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 13:43:06.021935 4319 raft_consensus.cc:348] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.022171 4319 raft_consensus.cc:374] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:06.022255 4319 raft_consensus.cc:729] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac79b63a036a4074b1550dcbb846a115, State: Initialized, Role: FOLLOWER
I20260927 13:43:06.022480 4319 consensus_queue.cc:260] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.022948 4319 ts_tablet_manager.cc:1428] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 13:43:06.024122 4318 tablet_bootstrap.cc:492] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: No bootstrap required, opened a new log
I20260927 13:43:06.024268 4318 ts_tablet_manager.cc:1397] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 13:43:06.024967 4318 raft_consensus.cc:348] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.025240 4318 raft_consensus.cc:374] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:06.025334 4318 raft_consensus.cc:729] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2b234d6784828986f9e4f3f2d2698, State: Initialized, Role: FOLLOWER
I20260927 13:43:06.025573 4318 consensus_queue.cc:260] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.026088 4318 ts_tablet_manager.cc:1428] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:06.247197 4325 raft_consensus.cc:482] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:06.247376 4325 raft_consensus.cc:504] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.247926 4325 leader_election.cc:288] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:06.248521 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a5193fd1b4464a10effe027e0cebc" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115" is_pre_election: true
I20260927 13:43:06.248564 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a5193fd1b4464a10effe027e0cebc" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698" is_pre_election: true
I20260927 13:43:06.248773 4025 raft_consensus.cc:2454] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:06.248776 3885 raft_consensus.cc:2454] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 0.
I20260927 13:43:06.249203 4092 leader_election.cc:302] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee2b234d6784828986f9e4f3f2d2698, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:06.249651 4325 raft_consensus.cc:2781] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:06.249770 4325 raft_consensus.cc:482] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:06.249857 4325 raft_consensus.cc:3037] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:06.252225 4325 raft_consensus.cc:504] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.252719 4325 leader_election.cc:288] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Requested vote from peers ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017), 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575)
I20260927 13:43:06.253129 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a5193fd1b4464a10effe027e0cebc" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac79b63a036a4074b1550dcbb846a115"
I20260927 13:43:06.253225 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a5193fd1b4464a10effe027e0cebc" candidate_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2b234d6784828986f9e4f3f2d2698"
I20260927 13:43:06.253327 3885 raft_consensus.cc:3037] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:06.253438 4025 raft_consensus.cc:3037] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:06.256593 3885 raft_consensus.cc:2454] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:06.256752 4025 raft_consensus.cc:2454] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ab148fcee4fb88bab92c78bd05c5f in term 1.
I20260927 13:43:06.257023 4092 leader_election.cc:302] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac79b63a036a4074b1550dcbb846a115, ee9ab148fcee4fb88bab92c78bd05c5f; no voters:
I20260927 13:43:06.257318 4325 raft_consensus.cc:2781] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:06.257465 4325 raft_consensus.cc:686] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Becoming Leader. State: Replica: ee9ab148fcee4fb88bab92c78bd05c5f, State: Running, Role: LEADER
I20260927 13:43:06.257740 4325 consensus_queue.cc:237] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } }
I20260927 13:43:06.260134 3724 catalog_manager.cc:5495] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195). New cstate: current_term: 1 leader_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee9ab148fcee4fb88bab92c78bd05c5f" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39781 } health_report { overall_health: HEALTHY } } }
I20260927 13:43:06.325201 4025 raft_consensus.cc:1261] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:06.325245 3885 raft_consensus.cc:1261] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Refusing update from remote peer ee9ab148fcee4fb88bab92c78bd05c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:06.325667 4325 consensus_queue.cc:1035] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee2b234d6784828986f9e4f3f2d2698" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:06.325894 4333 consensus_queue.cc:1035] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac79b63a036a4074b1550dcbb846a115" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:43:06.342605 4291 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 13:43:07.912073 3724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:07.912325 3724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:07.915663 3724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861c9dc76f504ad395e2155c92588a26: Sending DeleteTablet for 3 replicas of tablet ac1a5193fd1b4464a10effe027e0cebc
I20260927 13:43:07.916329 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a5193fd1b4464a10effe027e0cebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260927 13:43:07.916419 4138 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a5193fd1b4464a10effe027e0cebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43608
I20260927 13:43:07.916430 3999 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a5193fd1b4464a10effe027e0cebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58072
I20260927 13:43:07.917268 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3795
I20260927 13:43:07.927768 4365 tablet_replica.cc:330] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: stopping tablet replica
I20260927 13:43:07.928062 4365 raft_consensus.cc:2229] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:07.928349 4365 raft_consensus.cc:2258] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:07.928853 4363 tablet_replica.cc:330] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: stopping tablet replica
I20260927 13:43:07.929018 4363 raft_consensus.cc:2229] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:07.929255 4363 raft_consensus.cc:2258] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:07.929991 4365 ts_tablet_manager.cc:1905] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:07.931419 4363 ts_tablet_manager.cc:1905] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:07.935892 4364 tablet_replica.cc:330] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: stopping tablet replica
I20260927 13:43:07.936208 4364 raft_consensus.cc:2229] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:07.936635 4364 raft_consensus.cc:2258] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 13:43:07 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:07.937131 4365 ts_tablet_manager.cc:1918] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:07.937341 4365 log.cc:1196] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/ac1a5193fd1b4464a10effe027e0cebc
I20260927 13:43:07.937708 4364 ts_tablet_manager.cc:1905] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:07.937810 4365 ts_tablet_manager.cc:1939] T ac1a5193fd1b4464a10effe027e0cebc P 5ee2b234d6784828986f9e4f3f2d2698: Deleting consensus metadata
I20260927 13:43:07.938573 4363 ts_tablet_manager.cc:1918] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:07.938758 4363 log.cc:1196] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/ac1a5193fd1b4464a10effe027e0cebc
I20260927 13:43:07.938843 3710 catalog_manager.cc:4841] TS 5ee2b234d6784828986f9e4f3f2d2698 (127.1.237.194:37575): tablet ac1a5193fd1b4464a10effe027e0cebc (table test-table [id=0cb2f484c535474b94f8f32a851e7436]) successfully deleted
I20260927 13:43:07.939278 4363 ts_tablet_manager.cc:1939] T ac1a5193fd1b4464a10effe027e0cebc P ac79b63a036a4074b1550dcbb846a115: Deleting consensus metadata
I20260927 13:43:07.940155 3710 catalog_manager.cc:4841] TS ac79b63a036a4074b1550dcbb846a115 (127.1.237.193:37017): tablet ac1a5193fd1b4464a10effe027e0cebc (table test-table [id=0cb2f484c535474b94f8f32a851e7436]) successfully deleted
I20260927 13:43:07.943518 4364 ts_tablet_manager.cc:1918] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 13:43:07.943662 4364 log.cc:1196] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/ac1a5193fd1b4464a10effe027e0cebc
I20260927 13:43:07.944109 4364 ts_tablet_manager.cc:1939] T ac1a5193fd1b4464a10effe027e0cebc P ee9ab148fcee4fb88bab92c78bd05c5f: Deleting consensus metadata
I20260927 13:43:07.944440 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:07.918930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.1.237.193:37017)
0927 13:43:07.925572 (+ 6642us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:07.925580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:07.925604 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:43:07.925662 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:07.925688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:07.925798 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:07.925928 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:07.927960 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:07.929076 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:07.930878 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:07.931846 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:07.932032 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:07.938050 (+ 6018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:07.938072 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:07.938096 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:07.938138 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:07.941550 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:07.942303 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:07.942311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:07.942313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:07.942434 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:07.942865 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:07.942868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:07.942870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:07.943234 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:07.943435 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:07.943878 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 13:43:07.944026 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6473,"thread_start_us":6337,"threads_started":1}
W20260927 13:43:07.945071 3926 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40004
I20260927 13:43:07.945331 3710 catalog_manager.cc:4841] TS ee9ab148fcee4fb88bab92c78bd05c5f (127.1.237.195:39781): tablet ac1a5193fd1b4464a10effe027e0cebc (table test-table [id=0cb2f484c535474b94f8f32a851e7436]) successfully deleted
W20260927 13:43:07.945765 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3795: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:07.945935 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3795
I20260927 13:43:07.963099 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3935
Sep 27 13:43:07 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:07.977208 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:07.963752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.1.237.194:37575)
0927 13:43:07.964280 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:07.964284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:07.964300 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:43:07.964344 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:07.964347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:07.964404 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:07.964494 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:07.965724 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:07.966408 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:07.967639 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:07.968025 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:07.968113 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:07.972123 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:07.972134 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:07.972149 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:07.972178 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:07.974736 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:07.975395 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:07.975401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:07.975403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:07.975536 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:07.975898 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:07.975901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:07.975903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:07.976244 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:07.976451 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:07.976673 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 13:43:07.976814 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":217,"threads_started":1}
W20260927 13:43:07.977803 4065 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58298
W20260927 13:43:07.978408 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:07.978551 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3935
I20260927 13:43:08.002985 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4074
Sep 27 13:43:08 dist-test-slave-7g3l krb5kdc[3679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:08.016784 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:08.003672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.1.237.195:39781)
0927 13:43:08.004143 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:08.004147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:08.004163 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:43:08.004215 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:08.004218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:08.004278 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:08.004365 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:08.005561 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:08.006269 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:08.007449 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:08.007846 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:08.007933 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:08.011842 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:08.011855 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:08.011870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:08.011898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:08.014624 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:08.015130 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:08.015137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:08.015139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:08.015434 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:08.015719 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:08.015722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:08.015724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:08.016044 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:08.016226 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:08.016404 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 13:43:08.016546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
W20260927 13:43:08.017386 4204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50392
W20260927 13:43:08.017913 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:08.018061 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4074
I20260927 13:43:08.038625 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 3692
I20260927 13:43:08.039656 3764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:08.148772 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 3692
2026-09-27T13:43:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8055 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:08 dist-test-slave-7g3l krb5kdc[4373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:08 dist-test-slave-7g3l krb5kdc[4373](info): set up 2 sockets
Sep 27 13:43:08 dist-test-slave-7g3l krb5kdc[4373](info): commencing operation
krb5kdc: starting...
W20260927 13:43:09.203082 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:09 dist-test-slave-7g3l krb5kdc[4373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:09.368853 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36901
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:09.674806 4386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:09.675060 4386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:09.675144 4386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:09.682010 4386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:09.682098 4386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:09.682147 4386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:09.682200 4386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:09.682240 4386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:09.691705 4386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:09.692813 4386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:09.693948 4386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:09.705081 4392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:09.705166 4391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:09.705140 4394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:09.705914 4386 server_base.cc:1034] running on GCE node
I20260927 13:43:09.706516 4386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:09.707794 4386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:09.708981 4386 hybrid_clock.cc:648] HybridClock initialized: now 1790516589708938 us; error 38 us; skew 500 ppm
I20260927 13:43:09.709245 4386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:09 dist-test-slave-7g3l krb5kdc[4373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:09.712011 4386 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:09.713310 4386 webserver.cc:460] Webserver started at http://127.1.237.254:34423/ using document root <none> and password file <none>
I20260927 13:43:09.713881 4386 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:09.713970 4386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:09.714195 4386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:09.716828 4386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "5b2fa18f7fe34a62b4a988b19eec482d"
format_stamp: "Formatted at 2026-09-27 13:43:09 on dist-test-slave-7g3l"
server_key: "d6fd52729f5e040ad5514013e18506ea"
server_key_iv: "160d8cb8712687b83f3ef6506f9a26b4"
server_key_version: "encryptionkey@0"
I20260927 13:43:09.717483 4386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "5b2fa18f7fe34a62b4a988b19eec482d"
format_stamp: "Formatted at 2026-09-27 13:43:09 on dist-test-slave-7g3l"
server_key: "d6fd52729f5e040ad5514013e18506ea"
server_key_iv: "160d8cb8712687b83f3ef6506f9a26b4"
server_key_version: "encryptionkey@0"
I20260927 13:43:09.722018 4386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:09.724956 4401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:09.726101 4386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:09.726245 4386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "5b2fa18f7fe34a62b4a988b19eec482d"
format_stamp: "Formatted at 2026-09-27 13:43:09 on dist-test-slave-7g3l"
server_key: "d6fd52729f5e040ad5514013e18506ea"
server_key_iv: "160d8cb8712687b83f3ef6506f9a26b4"
server_key_version: "encryptionkey@0"
I20260927 13:43:09.726389 4386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:09.756703 4386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:09.760206 4386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:09.760425 4386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:09.777383 4386 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36901
I20260927 13:43:09.777403 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36901 every 8 connection(s)
I20260927 13:43:09.778565 4386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:09.781881 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:09.782431 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4386
I20260927 13:43:09.782667 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:09.783319 1975 external_mini_cluster.cc:1428] Setting key fcd77858b5742e20ff7b6a39cbaf2cc0
I20260927 13:43:09.795361 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d: Bootstrap starting.
Sep 27 13:43:09 dist-test-slave-7g3l krb5kdc[4373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:09.798076 4453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:09.799423 4453 log.cc:824] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:09.802441 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d: No bootstrap required, opened a new log
I20260927 13:43:09.805826 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:09.785173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.1.237.254:36901)
0927 13:43:09.785909 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:09.785943 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:09.785987 (+ 44us) server_negotiation.cc:407] Connection header received
0927 13:43:09.788131 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:09.788155 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:09.788600 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:09.789007 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:09.790182 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:09.792196 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:09.793456 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:09.794451 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:09.794608 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:09.798274 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:09.798284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:09.798300 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:09.798350 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:09.802060 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:09.802667 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:09.802676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:09.802683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:09.802890 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:09.803272 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:09.803279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:09.803286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:09.803784 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:09.804104 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:09.804558 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 13:43:09.804918 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":294,"threads_started":1}
I20260927 13:43:09.810057 4453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } }
I20260927 13:43:09.810374 4453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:09.810474 4453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2fa18f7fe34a62b4a988b19eec482d, State: Initialized, Role: FOLLOWER
I20260927 13:43:09.811069 4453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } }
I20260927 13:43:09.811386 4453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:09.811540 4453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:09.811710 4453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:09.813649 4453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } }
I20260927 13:43:09.814106 4453 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2fa18f7fe34a62b4a988b19eec482d; no voters:
I20260927 13:43:09.814590 4453 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:09.814700 4458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:09.815384 4458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [term 1 LEADER]: Becoming Leader. State: Replica: 5b2fa18f7fe34a62b4a988b19eec482d, State: Running, Role: LEADER
I20260927 13:43:09.815857 4458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } }
I20260927 13:43:09.816639 4453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:09.818483 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2fa18f7fe34a62b4a988b19eec482d. Latest consensus state: current_term: 1 leader_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } } }
I20260927 13:43:09.818910 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:09.818780 4459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2fa18f7fe34a62b4a988b19eec482d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36901 } } }
I20260927 13:43:09.819031 4459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:09.820627 4467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:09.825340 4467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:09.833412 4467 catalog_manager.cc:1347] Generated new cluster ID: 2529664d92e0497c83cc4f919f8531cc
I20260927 13:43:09.833540 4467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:09.868278 4467 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:09.869444 4467 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:09.882398 4467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2fa18f7fe34a62b4a988b19eec482d: Generated new TSK 0
I20260927 13:43:09.883354 4467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:09.932394 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:10.240581 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:10.240835 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:10.240921 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:10.247586 4479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:10.247668 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:10.247777 4479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:10.256711 4479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:10.257745 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:10.258852 4479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:10.270740 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:10.270782 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:10.270817 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:10.271481 4479 server_base.cc:1034] running on GCE node
I20260927 13:43:10.272027 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:10.272912 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:10.274127 4479 hybrid_clock.cc:648] HybridClock initialized: now 1790516590274081 us; error 43 us; skew 500 ppm
I20260927 13:43:10.274408 4479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:10 dist-test-slave-7g3l krb5kdc[4373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:10.277360 4479 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:10.278606 4479 webserver.cc:460] Webserver started at http://127.1.237.193:44635/ using document root <none> and password file <none>
I20260927 13:43:10.279233 4479 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:10.279330 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:10.279574 4479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:10.282087 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "3ed781bba73340c2bc1d41b5dadbf9bf"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "b3b51fa6a79d3cc5d3c4f59fb8d294eb"
server_key_iv: "0e1b610149ff2c640aed2e2481fb255a"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.282747 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "3ed781bba73340c2bc1d41b5dadbf9bf"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "b3b51fa6a79d3cc5d3c4f59fb8d294eb"
server_key_iv: "0e1b610149ff2c640aed2e2481fb255a"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.287348 4479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:43:10.290393 4494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.291661 4479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:10.291781 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "3ed781bba73340c2bc1d41b5dadbf9bf"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "b3b51fa6a79d3cc5d3c4f59fb8d294eb"
server_key_iv: "0e1b610149ff2c640aed2e2481fb255a"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.291934 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:10.309618 4479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:10.313092 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:10.313309 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:10.314110 4479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:10.315274 4479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:10.315343 4479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.315425 4479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:10.315466 4479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.346961 4479 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:34885
I20260927 13:43:10.347038 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:34885 every 8 connection(s)
I20260927 13:43:10.348079 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:10.348795 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4479
I20260927 13:43:10.349009 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:10.349720 1975 external_mini_cluster.cc:1428] Setting key 999f358c8db716eff9eedfb592f8bec1
Sep 27 13:43:10 dist-test-slave-7g3l krb5kdc[4373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:43:10.368918 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:10.350623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:46019 (local address 127.1.237.254:36901)
0927 13:43:10.351072 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:10.351077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:10.352049 (+ 972us) server_negotiation.cc:407] Connection header received
0927 13:43:10.353699 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:10.353706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:10.353784 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:10.353896 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:10.355191 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.356228 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.358495 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.358867 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.358948 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:10.362797 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:10.362805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:10.362813 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:10.362852 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:10.366264 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:10.366912 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:10.366920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:10.366926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:10.367088 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:10.367497 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:10.367504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:10.367509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:10.367854 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:10.367951 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:10.368521 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 13:43:10.368674 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:43:10.370012 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:10.351113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36901 (local address 127.1.237.193:46019)
0927 13:43:10.351867 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:10.351906 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:10.353458 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:10.354075 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:10.354082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:10.354520 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:10.355002 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:10.355016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.356370 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.356377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:10.358373 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:10.358388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.359078 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.359087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:10.359447 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:10.360099 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:10.360136 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:10.362610 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:10.366401 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:10.366406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:10.366415 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:10.366799 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:10.367228 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:10.367231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:10.367233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:10.367386 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:10.367963 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:10.367968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:10.368244 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 13:43:10.368472 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:43:10.368867 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":301,"threads_started":1}
I20260927 13:43:10.371429 4607 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36901
I20260927 13:43:10.371711 4607 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:10.372541 4607 heartbeater.cc:511] Master 127.1.237.254:36901 requested a full tablet report, sending...
I20260927 13:43:10.374796 4418 ts_manager.cc:194] Registered new tserver with Master: 3ed781bba73340c2bc1d41b5dadbf9bf (127.1.237.193:34885)
I20260927 13:43:10.377036 4418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:46019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:10.391673 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:10.694538 4613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:10.694826 4613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:10.694970 4613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:10.701711 4613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:10.701819 4613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:10.701988 4613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:10.710738 4613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:10.711869 4613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:10.713075 4613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:10.725236 4618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:10.725306 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:10.725274 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:10.726130 4613 server_base.cc:1034] running on GCE node
I20260927 13:43:10.726594 4613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:10.727463 4613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:10.728647 4613 hybrid_clock.cc:648] HybridClock initialized: now 1790516590728619 us; error 28 us; skew 500 ppm
I20260927 13:43:10.728924 4613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:10 dist-test-slave-7g3l krb5kdc[4373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:10.731957 4613 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:10.733232 4613 webserver.cc:460] Webserver started at http://127.1.237.194:36589/ using document root <none> and password file <none>
I20260927 13:43:10.733816 4613 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:10.733937 4613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:10.734244 4613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:10.736863 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "dd3990c3690e407aa638c62b93b8a26e"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "496960015dceedda47c47db30c50e422"
server_key_iv: "fa9bf149fd0523c1b2cbebbf7b2268bc"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.737560 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "dd3990c3690e407aa638c62b93b8a26e"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "496960015dceedda47c47db30c50e422"
server_key_iv: "fa9bf149fd0523c1b2cbebbf7b2268bc"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.742123 4613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 13:43:10.745616 4628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.746778 4613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:10.746951 4613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "dd3990c3690e407aa638c62b93b8a26e"
format_stamp: "Formatted at 2026-09-27 13:43:10 on dist-test-slave-7g3l"
server_key: "496960015dceedda47c47db30c50e422"
server_key_iv: "fa9bf149fd0523c1b2cbebbf7b2268bc"
server_key_version: "encryptionkey@0"
I20260927 13:43:10.747136 4613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:10.777504 4613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:10.781400 4613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:10.781668 4613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:10.782610 4613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:10.783875 4613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:10.783973 4613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.784126 4613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:10.784196 4613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:10.819185 4613 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:45859
I20260927 13:43:10.819233 4740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:45859 every 8 connection(s)
I20260927 13:43:10.820550 4613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:10.826327 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4613
I20260927 13:43:10.826521 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:10.827033 1975 external_mini_cluster.cc:1428] Setting key 63434a2b77e4c7f06dee5799267ace08
Sep 27 13:43:10 dist-test-slave-7g3l krb5kdc[4373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:10.839352 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:10.822967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:54405 (local address 127.1.237.254:36901)
0927 13:43:10.823144 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:10.823148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:10.824226 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 13:43:10.825933 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:10.825938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:10.826040 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:10.826161 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:10.827939 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.828781 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.830384 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.830774 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.830908 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:10.834168 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:10.834173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:10.834176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:10.834204 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:10.836590 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:10.837236 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:10.837241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:10.837244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:10.837369 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:10.837775 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:10.837780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:10.837783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:10.838017 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:10.838178 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:10.838762 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 13:43:10.838968 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:43:10.840221 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:10.823478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36901 (local address 127.1.237.194:54405)
0927 13:43:10.824090 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:10.824123 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:10.825690 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:10.826422 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:10.826434 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:10.827122 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:10.827789 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:10.827806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.828914 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.828920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:10.830257 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:10.830268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:10.830982 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:10.830990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:10.831268 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:10.831853 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:10.831876 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:10.833960 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:10.836720 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:10.836728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:10.836743 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:10.837119 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:10.837470 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:10.837476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:10.837479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:10.837645 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:10.838134 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:10.838141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:10.838439 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 13:43:10.838680 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 13:43:10.839195 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":256,"threads_started":1}
I20260927 13:43:10.841728 4741 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36901
I20260927 13:43:10.842013 4741 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:10.842648 4741 heartbeater.cc:511] Master 127.1.237.254:36901 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
I20260927 13:43:10.844097 4418 ts_manager.cc:194] Registered new tserver with Master: dd3990c3690e407aa638c62b93b8a26e (127.1.237.194:45859)
I20260927 13:43:10.844889 4418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:54405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:10.866693 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:11.166879 4746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:11.167124 4746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:11.167276 4746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:11.174042 4746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:11.174132 4746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:11.174239 4746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:11.184588 4746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:11.185614 4746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:11.186748 4746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:11.198932 4751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:11.199079 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:11.199239 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:11.200354 4746 server_base.cc:1034] running on GCE node
I20260927 13:43:11.200940 4746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:11.201725 4746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:11.202905 4746 hybrid_clock.cc:648] HybridClock initialized: now 1790516591202865 us; error 42 us; skew 500 ppm
I20260927 13:43:11.203245 4746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:11 dist-test-slave-7g3l krb5kdc[4373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:11.211143 4746 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:11.212518 4746 webserver.cc:460] Webserver started at http://127.1.237.195:36405/ using document root <none> and password file <none>
I20260927 13:43:11.213109 4746 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:11.213204 4746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:11.213425 4746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:11.216073 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "9572659e02a643c5911e5f783d765b27"
format_stamp: "Formatted at 2026-09-27 13:43:11 on dist-test-slave-7g3l"
server_key: "03a2bc332e728765f2cbe7746884f258"
server_key_iv: "22225ab1fa0d967033c22e6a77f77676"
server_key_version: "encryptionkey@0"
I20260927 13:43:11.216796 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "9572659e02a643c5911e5f783d765b27"
format_stamp: "Formatted at 2026-09-27 13:43:11 on dist-test-slave-7g3l"
server_key: "03a2bc332e728765f2cbe7746884f258"
server_key_iv: "22225ab1fa0d967033c22e6a77f77676"
server_key_version: "encryptionkey@0"
I20260927 13:43:11.221720 4746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:43:11.224783 4761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:11.226003 4746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:11.226166 4746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "9572659e02a643c5911e5f783d765b27"
format_stamp: "Formatted at 2026-09-27 13:43:11 on dist-test-slave-7g3l"
server_key: "03a2bc332e728765f2cbe7746884f258"
server_key_iv: "22225ab1fa0d967033c22e6a77f77676"
server_key_version: "encryptionkey@0"
I20260927 13:43:11.226344 4746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:11.239914 4746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:11.243868 4746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:11.244138 4746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:11.244997 4746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:11.246169 4746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:11.246240 4746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:11.246315 4746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:11.246368 4746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:11.277789 4746 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:45517
I20260927 13:43:11.277864 4873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:45517 every 8 connection(s)
I20260927 13:43:11.278990 4746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:11.280808 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4746
I20260927 13:43:11.280994 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:11.281509 1975 external_mini_cluster.cc:1428] Setting key 298896190458ad4fd8e1cd5e42aed872
Sep 27 13:43:11 dist-test-slave-7g3l krb5kdc[4373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:11.297442 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:11.281423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:57659 (local address 127.1.237.254:36901)
0927 13:43:11.281628 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:11.281636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:11.282841 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 13:43:11.284964 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:11.284968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:11.285034 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:11.285108 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:11.286376 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:11.287091 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:11.288544 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:11.288931 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:11.289042 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:11.292094 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:11.292099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:11.292102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:11.292132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:11.294523 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:11.295296 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:11.295302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:11.295306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:11.295421 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:11.295888 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:11.295893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:11.295896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:11.296223 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:11.296350 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:11.297074 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 13:43:11.297221 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:43:11.298220 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:11.281897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36901 (local address 127.1.237.195:57659)
0927 13:43:11.282660 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:11.282735 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:11.284740 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:11.285305 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:11.285312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:11.285745 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:11.286242 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:11.286258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:11.287253 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:11.287256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:11.288431 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:11.288442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:11.289096 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:11.289100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:11.289280 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:11.289862 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:11.289882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:11.291881 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:11.294645 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:11.294652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:11.294665 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:11.295201 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:11.295537 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:11.295541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:11.295545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:11.295746 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:11.296346 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:11.296352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:11.296728 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 13:43:11.296972 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:43:11.297276 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":288,"threads_started":1}
I20260927 13:43:11.299324 4874 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36901
I20260927 13:43:11.299592 4874 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:11.300231 4874 heartbeater.cc:511] Master 127.1.237.254:36901 requested a full tablet report, sending...
I20260927 13:43:11.301415 4418 ts_manager.cc:194] Registered new tserver with Master: 9572659e02a643c5911e5f783d765b27 (127.1.237.195:45517)
I20260927 13:43:11.302296 4418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:57659
I20260927 13:43:11.307996 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:43:11.336484 4609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:11.334350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.1.237.254:36901)
0927 13:43:11.334510 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:11.334515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:11.335008 (+ 493us) server_negotiation.cc:407] Connection header received
0927 13:43:11.335144 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:11.335148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:11.335249 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:11.335335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:11.336316 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35104: BlockingRecv error: recv got EOF from 127.0.0.1:35104 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:43:11.339404 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4479
W20260927 13:43:11.341846 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4479: Not authorized: Client connection negotiation failed: client connection to 127.1.237.193:34885: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:43:11.342018 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4479
I20260927 13:43:11.356652 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4613
W20260927 13:43:11.358690 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4613: Not authorized: Client connection negotiation failed: client connection to 127.1.237.194:45859: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:43:11.358810 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4613
I20260927 13:43:11.373101 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4746
W20260927 13:43:11.375550 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4746: Not authorized: Client connection negotiation failed: client connection to 127.1.237.195:45517: server requires authentication, but client does not have Kerberos credentials available
I20260927 13:43:11.375670 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4746
I20260927 13:43:11.389849 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4386
I20260927 13:43:11.390830 4448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:11.493324 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4386
2026-09-27T13:43:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3341 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:11 dist-test-slave-7g3l krb5kdc[4891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:11 dist-test-slave-7g3l krb5kdc[4891](info): set up 2 sockets
Sep 27 13:43:11 dist-test-slave-7g3l krb5kdc[4891](info): commencing operation
krb5kdc: starting...
W20260927 13:43:12.551265 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:12 dist-test-slave-7g3l krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:12.720340 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45737
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:13.016764 4904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:13.017138 4904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:13.017287 4904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:13.024933 4904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:13.025065 4904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:13.025141 4904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:13.025219 4904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:13.025285 4904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:13.034895 4904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:13.036204 4904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:13.037477 4904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:13.048532 4909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:13.048557 4912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:13.048547 4910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:13.049538 4904 server_base.cc:1034] running on GCE node
I20260927 13:43:13.050302 4904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:13.051630 4904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:13.052816 4904 hybrid_clock.cc:648] HybridClock initialized: now 1790516593052770 us; error 41 us; skew 500 ppm
I20260927 13:43:13.053120 4904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:13 dist-test-slave-7g3l krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:13.056139 4904 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:13.057401 4904 webserver.cc:460] Webserver started at http://127.1.237.254:34889/ using document root <none> and password file <none>
I20260927 13:43:13.058022 4904 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:13.058156 4904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:13.058457 4904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:13.061201 4904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "88e26cfcabdb42fd985c317443003e51"
format_stamp: "Formatted at 2026-09-27 13:43:13 on dist-test-slave-7g3l"
server_key: "29facaf2a25dc9ae08edbd2fae4fd732"
server_key_iv: "4da92de0c93e37e4878aab30194e732a"
server_key_version: "encryptionkey@0"
I20260927 13:43:13.061913 4904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "88e26cfcabdb42fd985c317443003e51"
format_stamp: "Formatted at 2026-09-27 13:43:13 on dist-test-slave-7g3l"
server_key: "29facaf2a25dc9ae08edbd2fae4fd732"
server_key_iv: "4da92de0c93e37e4878aab30194e732a"
server_key_version: "encryptionkey@0"
I20260927 13:43:13.066380 4904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:43:13.069414 4919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:13.070557 4904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:13.070722 4904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "88e26cfcabdb42fd985c317443003e51"
format_stamp: "Formatted at 2026-09-27 13:43:13 on dist-test-slave-7g3l"
server_key: "29facaf2a25dc9ae08edbd2fae4fd732"
server_key_iv: "4da92de0c93e37e4878aab30194e732a"
server_key_version: "encryptionkey@0"
I20260927 13:43:13.070900 4904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:13.096829 4904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:13.100621 4904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:13.100879 4904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:13.117352 4904 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:45737
I20260927 13:43:13.117388 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:45737 every 8 connection(s)
I20260927 13:43:13.118561 4904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:13.121893 4971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:13.125108 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 4904
I20260927 13:43:13.125317 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:13.125895 1975 external_mini_cluster.cc:1428] Setting key 03d0e0d88877e38422c797058465fd18
I20260927 13:43:13.132840 4971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51: Bootstrap starting.
I20260927 13:43:13.135457 4971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:13.136364 4971 log.cc:824] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:43:13 dist-test-slave-7g3l krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:13.138904 4971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51: No bootstrap required, opened a new log
I20260927 13:43:13.145591 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:13.127606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.1.237.254:45737)
0927 13:43:13.128288 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:13.128299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:13.128335 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:43:13.129764 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:13.129783 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:13.130179 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:13.130542 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:13.132022 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:13.133372 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:13.134423 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:13.134959 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:13.135171 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:13.138976 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:13.138988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:13.139005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:13.139046 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:13.141942 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:13.142520 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:13.142525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:13.142530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:13.142684 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:13.143043 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:13.143047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:13.143048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:13.143633 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:13.143942 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:13.144390 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:43:13.144746 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
I20260927 13:43:13.145958 4971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } }
I20260927 13:43:13.146380 4971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:13.146482 4971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e26cfcabdb42fd985c317443003e51, State: Initialized, Role: FOLLOWER
I20260927 13:43:13.147032 4971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } }
I20260927 13:43:13.147293 4971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:13.147446 4971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:13.147624 4971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:13.149379 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:43:13.150123 4971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } }
I20260927 13:43:13.150552 4971 leader_election.cc:302] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e26cfcabdb42fd985c317443003e51; no voters:
I20260927 13:43:13.151136 4971 leader_election.cc:288] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:13.151250 4976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:13.153458 4971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:13.157848 4976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [term 1 LEADER]: Becoming Leader. State: Replica: 88e26cfcabdb42fd985c317443003e51, State: Running, Role: LEADER
I20260927 13:43:13.158406 4976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } }
I20260927 13:43:13.161211 4977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e26cfcabdb42fd985c317443003e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } } }
I20260927 13:43:13.161559 4977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:13.161310 4978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e26cfcabdb42fd985c317443003e51. Latest consensus state: current_term: 1 leader_uuid: "88e26cfcabdb42fd985c317443003e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e26cfcabdb42fd985c317443003e51" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45737 } } }
I20260927 13:43:13.161687 4978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:13.163785 4989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:13.172161 4989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:13.181900 4989 catalog_manager.cc:1347] Generated new cluster ID: d9e1f98fe3434447bb16eb9c0c48a143
I20260927 13:43:13.182040 4989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:13.196256 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:13.185822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.1.237.254:45737)
0927 13:43:13.189192 (+ 3370us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:13.189202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:13.189781 (+ 579us) server_negotiation.cc:407] Connection header received
0927 13:43:13.189987 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:13.189996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:13.190070 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:13.190176 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:13.191125 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:13.191866 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:13.193161 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:13.193620 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:13.193755 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:13.194827 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:13.194843 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:13.194846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:13.195478 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:13.195541 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:13.195562 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:13.195679 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:13.195797 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 13:43:13.195895 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3170}
I20260927 13:43:13.210855 4989 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:13.212723 4989 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:13.222162 4989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e26cfcabdb42fd985c317443003e51: Generated new TSK 0
I20260927 13:43:13.223145 4989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:43:13.237023 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 4904
I20260927 13:43:13.238462 4966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:13.324954 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 4904
2026-09-27T13:43:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1825 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:13 dist-test-slave-7g3l krb5kdc[5005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:13 dist-test-slave-7g3l krb5kdc[5005](info): set up 2 sockets
Sep 27 13:43:13 dist-test-slave-7g3l krb5kdc[5005](info): commencing operation
krb5kdc: starting...
W20260927 13:43:14.371404 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:14 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:14.542112 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41185
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:14.833755 5018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:14.834028 5018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:14.834177 5018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:14.840595 5018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:14.840715 5018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:14.840814 5018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:14.840909 5018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:14.840976 5018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:14.849798 5018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:14.850948 5018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:14.852154 5018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:14.862658 5023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:14.862718 5024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:14.862758 5026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:14.863523 5018 server_base.cc:1034] running on GCE node
I20260927 13:43:14.864199 5018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:14.865350 5018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:14.866533 5018 hybrid_clock.cc:648] HybridClock initialized: now 1790516594866496 us; error 29 us; skew 500 ppm
I20260927 13:43:14.866827 5018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:14 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516594, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:14.869620 5018 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:14.870851 5018 webserver.cc:460] Webserver started at http://127.1.237.254:38325/ using document root <none> and password file <none>
I20260927 13:43:14.871505 5018 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:14.871627 5018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:14.871903 5018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:14.874481 5018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "7448b1fb8bc547caa6c5c075467c8e3d"
format_stamp: "Formatted at 2026-09-27 13:43:14 on dist-test-slave-7g3l"
server_key: "5d394e008f450b5a84aea3537f3c7132"
server_key_iv: "ea539e8cfe19edc3db62d2d450723ac1"
server_key_version: "encryptionkey@0"
I20260927 13:43:14.875124 5018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "7448b1fb8bc547caa6c5c075467c8e3d"
format_stamp: "Formatted at 2026-09-27 13:43:14 on dist-test-slave-7g3l"
server_key: "5d394e008f450b5a84aea3537f3c7132"
server_key_iv: "ea539e8cfe19edc3db62d2d450723ac1"
server_key_version: "encryptionkey@0"
I20260927 13:43:14.879504 5018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:43:14.882647 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:14.883909 5018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:14.884101 5018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "7448b1fb8bc547caa6c5c075467c8e3d"
format_stamp: "Formatted at 2026-09-27 13:43:14 on dist-test-slave-7g3l"
server_key: "5d394e008f450b5a84aea3537f3c7132"
server_key_iv: "ea539e8cfe19edc3db62d2d450723ac1"
server_key_version: "encryptionkey@0"
I20260927 13:43:14.884276 5018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:14.910813 5018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:14.914192 5018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:14.914439 5018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:14.930925 5018 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:41185
I20260927 13:43:14.930961 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:41185 every 8 connection(s)
I20260927 13:43:14.932145 5018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:14.935525 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5018
I20260927 13:43:14.935585 5085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:14.935739 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:14.936477 1975 external_mini_cluster.cc:1428] Setting key 7713642aa56f2170ae84897955165b18
I20260927 13:43:14.949800 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d: Bootstrap starting.
Sep 27 13:43:14 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:14.953265 5085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:14.954459 5085 log.cc:824] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:14.957588 5085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d: No bootstrap required, opened a new log
I20260927 13:43:14.959010 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:14.938252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.1.237.254:41185)
0927 13:43:14.938949 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:14.938959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:14.938993 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:43:14.940870 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:14.940888 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:14.941264 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:14.941574 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:14.943206 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:14.944399 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:14.946112 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:14.946768 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:14.946892 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:14.951217 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:14.951228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:14.951259 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:14.951309 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:14.954835 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:14.955710 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:14.955715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:14.955720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:14.955921 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:14.956293 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:14.956297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:14.956300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:14.956917 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:14.957199 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:14.957438 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:43:14.957808 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":344,"threads_started":1}
I20260927 13:43:14.965332 5085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } }
I20260927 13:43:14.965606 5085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:14.965672 5085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7448b1fb8bc547caa6c5c075467c8e3d, State: Initialized, Role: FOLLOWER
I20260927 13:43:14.966128 5085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } }
I20260927 13:43:14.966337 5085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:14.966462 5085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:14.966614 5085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:14.968758 5085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } }
I20260927 13:43:14.969178 5085 leader_election.cc:302] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7448b1fb8bc547caa6c5c075467c8e3d; no voters:
I20260927 13:43:14.969834 5085 leader_election.cc:288] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:14.969924 5090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:14.970685 5090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [term 1 LEADER]: Becoming Leader. State: Replica: 7448b1fb8bc547caa6c5c075467c8e3d, State: Running, Role: LEADER
I20260927 13:43:14.971252 5090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } }
I20260927 13:43:14.971866 5085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:14.975870 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7448b1fb8bc547caa6c5c075467c8e3d. Latest consensus state: current_term: 1 leader_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } } }
I20260927 13:43:14.976040 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7448b1fb8bc547caa6c5c075467c8e3d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41185 } } }
I20260927 13:43:14.976356 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:14.976356 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:14.978483 5098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:14.983774 5098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:14.992300 5098 catalog_manager.cc:1347] Generated new cluster ID: 2cfcaeb872384aa19ac5f4ab4a3b4f82
I20260927 13:43:14.992410 5098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:15.022004 5098 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:15.023289 5098 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:15.039142 5098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7448b1fb8bc547caa6c5c075467c8e3d: Generated new TSK 0
I20260927 13:43:15.039983 5098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:15.096540 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:15.410173 5111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:15.410418 5111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:15.410504 5111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:15.417136 5111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:15.417228 5111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:15.417387 5111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:15.426374 5111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:15.427500 5111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:15.428643 5111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:15.442853 5116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:15.443037 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:15.443459 5119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:15.443671 5111 server_base.cc:1034] running on GCE node
I20260927 13:43:15.444442 5111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:15.445573 5111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:15.446806 5111 hybrid_clock.cc:648] HybridClock initialized: now 1790516595446715 us; error 87 us; skew 500 ppm
I20260927 13:43:15.447113 5111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:15 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:15.450166 5111 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:15.451594 5111 webserver.cc:460] Webserver started at http://127.1.237.193:41137/ using document root <none> and password file <none>
I20260927 13:43:15.452243 5111 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:15.452370 5111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:15.452662 5111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:15.455412 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "a682795954994142b62783470a0b620f"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "1ed802b60d6f8070254f13e7a1504d10"
server_key_iv: "8663b2314a814acdb56e56c8c9efb0cf"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.456223 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "a682795954994142b62783470a0b620f"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "1ed802b60d6f8070254f13e7a1504d10"
server_key_iv: "8663b2314a814acdb56e56c8c9efb0cf"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.461299 5111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:43:15.464381 5126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.465543 5111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:15.465701 5111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a682795954994142b62783470a0b620f"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "1ed802b60d6f8070254f13e7a1504d10"
server_key_iv: "8663b2314a814acdb56e56c8c9efb0cf"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.465878 5111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:15.493712 5111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:15.497738 5111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:15.498027 5111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:15.498884 5111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:15.500181 5111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:15.500250 5111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.500412 5111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:15.500473 5111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.531797 5111 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37817
I20260927 13:43:15.531862 5238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37817 every 8 connection(s)
I20260927 13:43:15.533349 5111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:15.542795 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5111
I20260927 13:43:15.542975 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:15.543634 1975 external_mini_cluster.cc:1428] Setting key 34f2289c2745aa5a0f6539cd8b7a673a
Sep 27 13:43:15 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:15.553931 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:15.536163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:47733 (local address 127.1.237.254:41185)
0927 13:43:15.536651 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:15.536658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:15.537758 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 13:43:15.539877 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:15.539881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:15.539941 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:15.540034 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:15.541489 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:15.542138 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:15.543755 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:15.544142 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:15.544266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:15.547818 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:15.547826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:15.547833 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:15.547874 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:15.551003 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:15.551919 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:15.551922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:15.551924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:15.552036 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:15.552481 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:15.552484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:15.552486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:15.552711 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:15.552808 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:15.553560 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 13:43:15.553697 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260927 13:43:15.555240 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:15.536747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41185 (local address 127.1.237.193:47733)
0927 13:43:15.537586 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:15.537629 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:15.539647 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:15.540244 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:15.540251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:15.540714 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:15.541358 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:15.541374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:15.542258 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:15.542261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:15.543602 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:15.543613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:15.544363 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:15.544370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:15.544628 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:15.545236 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:15.545256 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:15.547625 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:15.551128 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:15.551137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:15.551184 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:15.551761 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:15.552150 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:15.552157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:15.552164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:15.552379 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:15.552821 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:15.552831 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:15.553223 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 13:43:15.553476 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 13:43:15.553961 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":323,"threads_started":1}
I20260927 13:43:15.556604 5239 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41185
I20260927 13:43:15.556864 5239 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:15.557487 5239 heartbeater.cc:511] Master 127.1.237.254:41185 requested a full tablet report, sending...
I20260927 13:43:15.559612 5050 ts_manager.cc:194] Registered new tserver with Master: a682795954994142b62783470a0b620f (127.1.237.193:37817)
I20260927 13:43:15.561597 5050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:47733
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:15.583937 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:15.888402 5245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:15.888682 5245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:15.888836 5245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:15.895704 5245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:15.895826 5245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:15.895983 5245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:15.905086 5245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:15.906198 5245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:15.907544 5245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:15.919191 5250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:15.919350 5253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:15.919533 5251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:15.920495 5245 server_base.cc:1034] running on GCE node
I20260927 13:43:15.921020 5245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:15.921833 5245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:15.923055 5245 hybrid_clock.cc:648] HybridClock initialized: now 1790516595923017 us; error 48 us; skew 500 ppm
I20260927 13:43:15.923369 5245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:15 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:15.926260 5245 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:15.927621 5245 webserver.cc:460] Webserver started at http://127.1.237.194:46153/ using document root <none> and password file <none>
I20260927 13:43:15.928232 5245 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:15.928363 5245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:15.928660 5245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:15.931313 5245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "70e260d0f67049b9871cafa76c7e0bc2"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "758cb07022d0e97daa9e992832770b27"
server_key_iv: "6cd7e0443fefdecb6b42a3e60ed921b7"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.932044 5245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "70e260d0f67049b9871cafa76c7e0bc2"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "758cb07022d0e97daa9e992832770b27"
server_key_iv: "6cd7e0443fefdecb6b42a3e60ed921b7"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.936693 5245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:15.939905 5260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.941167 5245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:15.941316 5245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "70e260d0f67049b9871cafa76c7e0bc2"
format_stamp: "Formatted at 2026-09-27 13:43:15 on dist-test-slave-7g3l"
server_key: "758cb07022d0e97daa9e992832770b27"
server_key_iv: "6cd7e0443fefdecb6b42a3e60ed921b7"
server_key_version: "encryptionkey@0"
I20260927 13:43:15.941488 5245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:15.952606 5245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:15.956451 5245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:15.956733 5245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:15.957551 5245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:15.958748 5245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:15.958845 5245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.958987 5245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:15.959060 5245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:15.991679 5245 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:34951
I20260927 13:43:15.991722 5372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:34951 every 8 connection(s)
I20260927 13:43:15.992870 5245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:15.997818 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5245
I20260927 13:43:15.997991 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:15.998399 1975 external_mini_cluster.cc:1428] Setting key 5fa69a5a08fac35780b4b302185d210d
Sep 27 13:43:15 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:16.013784 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:15.995270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:36825 (local address 127.1.237.254:41185)
0927 13:43:15.995436 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:15.995441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:15.996645 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 13:43:15.998448 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:15.998453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:15.998542 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:15.998645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.000340 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.001412 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.002800 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.003562 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.003694 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.007039 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.007048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.007056 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.007097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.010640 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.011328 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.011336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.011342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.011491 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.011874 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.011881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.011887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.012265 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.012430 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.013142 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.013360 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:43:16.014351 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:15.995727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41185 (local address 127.1.237.194:36825)
0927 13:43:15.996449 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:15.996509 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:15.998210 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:15.998960 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:15.998991 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:15.999555 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:16.000142 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:16.000183 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.001561 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.001565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:16.002659 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:16.002673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.003735 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.003740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:16.003918 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.004542 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:16.004562 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:16.006818 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:16.010767 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:16.010773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:16.010783 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:16.011223 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:16.011597 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:16.011601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:16.011603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:16.011757 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:16.012416 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:16.012422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:16.012692 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 13:43:16.012996 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 13:43:16.013351 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260927 13:43:16.015761 5373 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41185
I20260927 13:43:16.016072 5373 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:16.016858 5373 heartbeater.cc:511] Master 127.1.237.254:41185 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
I20260927 13:43:16.018282 5050 ts_manager.cc:194] Registered new tserver with Master: 70e260d0f67049b9871cafa76c7e0bc2 (127.1.237.194:34951)
I20260927 13:43:16.019009 5050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:36825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:16.040203 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:16.332170 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:16.332453 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:16.332631 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:16.339253 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:16.339373 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:16.339540 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:16.348423 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:16.349574 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:16.350746 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:16.362739 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:16.362787 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:16.362828 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:16.363765 5378 server_base.cc:1034] running on GCE node
I20260927 13:43:16.364321 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:16.365140 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:16.366361 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790516596366379 us; error 99 us; skew 500 ppm
I20260927 13:43:16.366639 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:16.373030 5378 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:16.374737 5378 webserver.cc:460] Webserver started at http://127.1.237.195:40553/ using document root <none> and password file <none>
I20260927 13:43:16.375406 5378 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:16.375558 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:16.375828 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:16.378408 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "0df7bc2488ec4b6f93cb3b14201d6922"
format_stamp: "Formatted at 2026-09-27 13:43:16 on dist-test-slave-7g3l"
server_key: "c458ee145c3881a2448671c531598d24"
server_key_iv: "1bb2f80836a8ac777345e964ab932d2e"
server_key_version: "encryptionkey@0"
I20260927 13:43:16.379225 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "0df7bc2488ec4b6f93cb3b14201d6922"
format_stamp: "Formatted at 2026-09-27 13:43:16 on dist-test-slave-7g3l"
server_key: "c458ee145c3881a2448671c531598d24"
server_key_iv: "1bb2f80836a8ac777345e964ab932d2e"
server_key_version: "encryptionkey@0"
I20260927 13:43:16.383849 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:16.386981 5393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:16.388228 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:16.388386 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "0df7bc2488ec4b6f93cb3b14201d6922"
format_stamp: "Formatted at 2026-09-27 13:43:16 on dist-test-slave-7g3l"
server_key: "c458ee145c3881a2448671c531598d24"
server_key_iv: "1bb2f80836a8ac777345e964ab932d2e"
server_key_version: "encryptionkey@0"
I20260927 13:43:16.388545 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:16.408425 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:16.412314 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:16.412595 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:16.413436 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:16.414562 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:16.414628 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:16.414783 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:16.414830 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:16.445891 5378 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:36455
I20260927 13:43:16.445927 5505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:36455 every 8 connection(s)
I20260927 13:43:16.447043 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:16.453686 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5378
I20260927 13:43:16.453855 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:16.454398 1975 external_mini_cluster.cc:1428] Setting key ee72c43e7612ab886eac5bef1b73a70e
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:16.465986 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.449428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:55457 (local address 127.1.237.254:41185)
0927 13:43:16.449588 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:16.449592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:16.450656 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 13:43:16.452450 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:16.452457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:16.452548 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:16.452657 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.454394 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.455338 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.457245 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.457642 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.457731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.460861 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.460865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.460868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.460896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.463375 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.463999 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.464005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.464008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.464119 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.464488 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.464494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.464497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.464733 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.464871 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.465461 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.465669 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:43:16.466749 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.449886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41185 (local address 127.1.237.195:55457)
0927 13:43:16.450510 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:16.450552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:16.452203 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:16.452873 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:16.452881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:16.453454 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:16.454211 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:16.454235 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.455503 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.455508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:16.457115 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:16.457131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.457806 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.457810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:16.457984 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.458541 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:16.458560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:16.460679 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:16.463497 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:16.463505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:16.463517 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:16.463887 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:16.464222 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:16.464228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:16.464231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:16.464387 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:16.464859 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:16.464866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:16.465145 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 13:43:16.465379 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 13:43:16.465806 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":241,"threads_started":1}
I20260927 13:43:16.467998 5506 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41185
I20260927 13:43:16.468292 5506 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:16.469074 5506 heartbeater.cc:511] Master 127.1.237.254:41185 requested a full tablet report, sending...
I20260927 13:43:16.470379 5050 ts_manager.cc:194] Registered new tserver with Master: 0df7bc2488ec4b6f93cb3b14201d6922 (127.1.237.195:36455)
I20260927 13:43:16.471146 5050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:55457
I20260927 13:43:16.480718 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:16.523458 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.511268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.1.237.254:41185)
0927 13:43:16.511478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:16.511487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:16.511815 (+ 328us) server_negotiation.cc:407] Connection header received
0927 13:43:16.511959 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:16.511963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:16.512033 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:16.512115 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.513300 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.513989 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.515146 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.515599 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.515691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.518986 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.518991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.518993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.519022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.521425 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.521913 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.521916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.521918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.522024 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.522316 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.522319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.522321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.522603 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.522763 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.522966 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.523105 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 13:43:16.523988 5050 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42536
I20260927 13:43:16.525276 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5111
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:16.540570 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.526237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.1.237.193:37817)
0927 13:43:16.526713 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:16.526720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:16.526744 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:43:16.526793 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:16.526799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:16.526998 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:16.527116 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.528171 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.529096 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.530138 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.530773 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.530863 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.535110 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.535120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.535132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.535218 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.538003 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.538603 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.538613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.538616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.538723 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.539096 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.539101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.539104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.539447 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.539786 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.539999 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.540239 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
W20260927 13:43:16.541677 5234 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51198
W20260927 13:43:16.542667 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5111: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:16.542830 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5111
I20260927 13:43:16.558075 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5245
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:16.571348 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.558783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.1.237.194:34951)
0927 13:43:16.559317 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:16.559323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:16.559345 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:43:16.559393 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:16.559398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:16.559566 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:16.559674 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.560731 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.561629 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.562664 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.563126 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.563311 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.566728 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.566736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.566746 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.566772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.569190 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.569702 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.569707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.569709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.569808 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.570155 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.570158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.570160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.570457 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.570642 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.570878 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.571041 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":238,"threads_started":1}
W20260927 13:43:16.572482 5368 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42072
W20260927 13:43:16.573557 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:16.573699 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5245
I20260927 13:43:16.589128 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5378
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:16.603111 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:16.589898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.1.237.195:36455)
0927 13:43:16.590455 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:16.590461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:16.590486 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:43:16.590536 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:16.590542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:16.590708 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:16.590814 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:16.592046 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.593111 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.594187 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:16.594678 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:16.594787 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:16.598432 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:16.598449 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:16.598459 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:16.598488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:16.600989 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.601540 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.601545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.601547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.601652 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:16.601918 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:16.601921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:16.601923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:16.602178 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:16.602367 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:16.602592 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 13:43:16.602757 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
W20260927 13:43:16.604262 5501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43522
W20260927 13:43:16.605211 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:43:16.605382 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5378
I20260927 13:43:16.619831 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5018
I20260927 13:43:16.620759 5080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:16.719403 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5018
2026-09-27T13:43:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3399 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5523](info): set up 2 sockets
Sep 27 13:43:16 dist-test-slave-7g3l krb5kdc[5523](info): commencing operation
krb5kdc: starting...
W20260927 13:43:17.759418 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:17 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:17.932708 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45859
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:18.232399 5536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:18.232651 5536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:18.232734 5536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:18.239306 5536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:18.239390 5536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:18.239442 5536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:18.239495 5536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:18.239529 5536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:18.248438 5536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:18.249536 5536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:18.250681 5536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:18.261451 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:18.261513 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:18.261538 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:18.262303 5536 server_base.cc:1034] running on GCE node
I20260927 13:43:18.262975 5536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:18.264195 5536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:18.265434 5536 hybrid_clock.cc:648] HybridClock initialized: now 1790516598265406 us; error 41 us; skew 500 ppm
I20260927 13:43:18.265743 5536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:18 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516598, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:18.268683 5536 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:18.269903 5536 webserver.cc:460] Webserver started at http://127.1.237.254:34347/ using document root <none> and password file <none>
I20260927 13:43:18.270510 5536 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:18.270648 5536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:18.270936 5536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:18.273563 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "07e8f65875384a9caedc2d3955710e71"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "ad4502fdaccff94b97be13399f09668a"
server_key_iv: "226fb07adb5dd27b99bc2fe2391b3996"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.274281 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "07e8f65875384a9caedc2d3955710e71"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "ad4502fdaccff94b97be13399f09668a"
server_key_iv: "226fb07adb5dd27b99bc2fe2391b3996"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.278676 5536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 13:43:18.281719 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:18.282860 5536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:18.283006 5536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "07e8f65875384a9caedc2d3955710e71"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "ad4502fdaccff94b97be13399f09668a"
server_key_iv: "226fb07adb5dd27b99bc2fe2391b3996"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.283214 5536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:18.297618 5536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:18.301481 5536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:18.301745 5536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:18.318641 5536 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:45859
I20260927 13:43:18.318666 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:45859 every 8 connection(s)
I20260927 13:43:18.319885 5536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:18.323139 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:18.325807 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5536
I20260927 13:43:18.326026 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:18.326753 1975 external_mini_cluster.cc:1428] Setting key 876f28d786e5d361bd943913b5234ca0
I20260927 13:43:18.335089 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71: Bootstrap starting.
I20260927 13:43:18.338480 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:18.339747 5603 log.cc:824] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:43:18 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:18.342428 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71: No bootstrap required, opened a new log
I20260927 13:43:18.348165 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:18.328628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.1.237.254:45859)
0927 13:43:18.329306 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:18.329316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:18.329352 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:43:18.330836 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:18.330853 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:18.331250 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:18.331566 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:18.333455 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.334632 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.336335 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.336931 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.337045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:18.341570 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:18.341576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:18.341588 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:18.341615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:18.344529 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:18.345270 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:18.345276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:18.345280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:18.345430 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:18.345802 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:18.345805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:18.345808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:18.346265 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:18.346463 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:18.346872 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 13:43:18.347227 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
I20260927 13:43:18.351117 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } }
I20260927 13:43:18.351508 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:18.351612 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e8f65875384a9caedc2d3955710e71, State: Initialized, Role: FOLLOWER
I20260927 13:43:18.352128 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } }
I20260927 13:43:18.352353 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:18.352509 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:18.352666 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:18.354751 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } }
I20260927 13:43:18.355257 5603 leader_election.cc:302] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e8f65875384a9caedc2d3955710e71; no voters:
I20260927 13:43:18.355809 5603 leader_election.cc:288] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:18.355895 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:18.356623 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [term 1 LEADER]: Becoming Leader. State: Replica: 07e8f65875384a9caedc2d3955710e71, State: Running, Role: LEADER
I20260927 13:43:18.357131 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } }
I20260927 13:43:18.357764 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:18.359881 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e8f65875384a9caedc2d3955710e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } } }
I20260927 13:43:18.360193 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:18.359963 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e8f65875384a9caedc2d3955710e71. Latest consensus state: current_term: 1 leader_uuid: "07e8f65875384a9caedc2d3955710e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65875384a9caedc2d3955710e71" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45859 } } }
I20260927 13:43:18.360595 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:18.364005 5615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:18.370419 5615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:18.380751 5615 catalog_manager.cc:1347] Generated new cluster ID: 9a40fa121bbc48f09771fa8373fb3e69
I20260927 13:43:18.380841 5615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:18.391137 5615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:18.392319 5615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:18.399537 5615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e8f65875384a9caedc2d3955710e71: Generated new TSK 0
I20260927 13:43:18.400300 5615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:18.455719 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:18.780233 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:18.780478 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:18.780617 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:18.787421 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:18.787530 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:18.787703 5629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:18.796944 5629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:18.798030 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:18.799223 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:18.811358 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:18.811491 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:18.811512 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:18.812408 5629 server_base.cc:1034] running on GCE node
I20260927 13:43:18.812968 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:18.813844 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:18.815056 5629 hybrid_clock.cc:648] HybridClock initialized: now 1790516598815024 us; error 47 us; skew 500 ppm
I20260927 13:43:18.815366 5629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:18 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516598, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:18.818380 5629 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:18.819676 5629 webserver.cc:460] Webserver started at http://127.1.237.193:37233/ using document root <none> and password file <none>
I20260927 13:43:18.820232 5629 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:18.820325 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:18.820591 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:18.823300 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "2869e0b5698f44e29ac5860b45315d6d"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "a89035a223fdb7871dbc8ebd52355ebd"
server_key_iv: "d943dec5c51b2af4c6c11841f71c5de3"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.823975 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "2869e0b5698f44e29ac5860b45315d6d"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "a89035a223fdb7871dbc8ebd52355ebd"
server_key_iv: "d943dec5c51b2af4c6c11841f71c5de3"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.828375 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 13:43:18.831517 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:18.832742 5629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:43:18.832885 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "2869e0b5698f44e29ac5860b45315d6d"
format_stamp: "Formatted at 2026-09-27 13:43:18 on dist-test-slave-7g3l"
server_key: "a89035a223fdb7871dbc8ebd52355ebd"
server_key_iv: "d943dec5c51b2af4c6c11841f71c5de3"
server_key_version: "encryptionkey@0"
I20260927 13:43:18.833065 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:18.849200 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:18.853183 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:18.853446 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:18.854249 5629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:18.855497 5629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:18.855566 5629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:18.855733 5629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:18.855782 5629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:18.886448 5629 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:34411
I20260927 13:43:18.886495 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:34411 every 8 connection(s)
I20260927 13:43:18.887727 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:18.892171 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5629
I20260927 13:43:18.892370 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:18.892984 1975 external_mini_cluster.cc:1428] Setting key 82ba1f8809d79dad3796a497781f7497
Sep 27 13:43:18 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516598, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:18.907585 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:18.890341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:58773 (local address 127.1.237.254:45859)
0927 13:43:18.890769 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:18.890773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:18.891677 (+ 904us) server_negotiation.cc:407] Connection header received
0927 13:43:18.893509 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:18.893522 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:18.893600 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:18.893706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:18.895217 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.896376 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.897733 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.898436 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.898570 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:18.902185 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:18.902190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:18.902193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:18.902220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:18.904587 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:18.905465 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:18.905469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:18.905471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:18.905581 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:18.906033 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:18.906036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:18.906037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:18.906260 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:18.906369 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:18.907145 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 13:43:18.907343 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":241,"threads_started":1}
I20260927 13:43:18.908880 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:18.890825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45859 (local address 127.1.237.193:58773)
0927 13:43:18.891524 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:18.891575 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:18.893271 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:18.893909 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:18.893917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:18.894452 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:18.895033 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:18.895050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.896510 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.896513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:18.897581 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:18.897592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:18.898705 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:18.898714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:18.898921 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:18.899579 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:18.899638 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:18.901951 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:18.904755 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:18.904782 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:18.904798 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:18.905364 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:18.905685 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:18.905693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:18.905699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:18.905920 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:18.906387 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:18.906397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:18.906800 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 13:43:18.907093 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:43:18.907624 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260927 13:43:18.910254 5757 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45859
I20260927 13:43:18.910527 5757 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:18.911146 5757 heartbeater.cc:511] Master 127.1.237.254:45859 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:43:18.913161 5568 ts_manager.cc:194] Registered new tserver with Master: 2869e0b5698f44e29ac5860b45315d6d (127.1.237.193:34411)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:43:18.914808 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:58773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:18.933830 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:19.237991 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:19.238253 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:19.238405 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:19.245182 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:19.245325 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:19.245509 5763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:19.254525 5763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:19.255724 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:19.256916 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:19.269174 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:19.269196 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:19.269395 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:19.269943 5763 server_base.cc:1034] running on GCE node
I20260927 13:43:19.270506 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:19.271373 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:19.272588 5763 hybrid_clock.cc:648] HybridClock initialized: now 1790516599272548 us; error 54 us; skew 500 ppm
I20260927 13:43:19.272882 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:19.275695 5763 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:19.277001 5763 webserver.cc:460] Webserver started at http://127.1.237.194:35071/ using document root <none> and password file <none>
I20260927 13:43:19.277634 5763 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:19.277770 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:19.278070 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:19.280723 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "d839ce57d561493f95e18971a8b6fd7e"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "eb4833c40b203c882a0340e68794531a"
server_key_iv: "2682f0441144c14d8be087b1c75ad8a5"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.281428 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "d839ce57d561493f95e18971a8b6fd7e"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "eb4833c40b203c882a0340e68794531a"
server_key_iv: "2682f0441144c14d8be087b1c75ad8a5"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.286017 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:43:19.289144 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.290297 5763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:19.290459 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "d839ce57d561493f95e18971a8b6fd7e"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "eb4833c40b203c882a0340e68794531a"
server_key_iv: "2682f0441144c14d8be087b1c75ad8a5"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.290647 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:19.304123 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:19.307789 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:19.308043 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:19.308845 5763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:19.309964 5763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:19.310061 5763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.310201 5763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:19.310285 5763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.344080 5763 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:36073
I20260927 13:43:19.344154 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:36073 every 8 connection(s)
I20260927 13:43:19.345230 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:19.348109 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5763
I20260927 13:43:19.348309 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:19.348766 1975 external_mini_cluster.cc:1428] Setting key c16219ee210a16a200296accadbe7930
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:19.364886 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.347598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:34593 (local address 127.1.237.254:45859)
0927 13:43:19.347770 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:19.347775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:19.348977 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 13:43:19.350882 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:19.350887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:19.350967 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:19.351088 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:19.352450 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.353573 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.354972 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.355610 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.355700 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.359666 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:19.359671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:19.359674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:19.359702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:19.362172 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.362810 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.362817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.362825 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.362937 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.363382 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.363389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.363392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.363636 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:19.363789 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:19.364368 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 13:43:19.364578 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:43:19.365772 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.348018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45859 (local address 127.1.237.194:34593)
0927 13:43:19.348826 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:19.348869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:19.350647 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:19.351316 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:19.351323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:19.351782 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:19.352277 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:19.352292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.353709 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.353713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:19.354839 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:19.354850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.355812 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.355820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:19.356112 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.356807 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:19.356837 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:19.359443 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:19.362307 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:19.362315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:19.362328 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:19.362695 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:19.363043 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:19.363048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:19.363051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:19.363281 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:19.363751 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:19.363760 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:19.364057 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 13:43:19.364282 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 13:43:19.364721 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
I20260927 13:43:19.367108 5891 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45859
I20260927 13:43:19.367411 5891 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:19.368155 5891 heartbeater.cc:511] Master 127.1.237.254:45859 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
I20260927 13:43:19.369529 5568 ts_manager.cc:194] Registered new tserver with Master: d839ce57d561493f95e18971a8b6fd7e (127.1.237.194:36073)
I20260927 13:43:19.370266 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:34593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:19.388348 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:19.693439 5896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:19.693689 5896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:19.693830 5896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:19.700770 5896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:19.700892 5896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:19.701010 5896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:19.709975 5896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:19.711126 5896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:19.712345 5896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:19.724512 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:19.724565 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:19.724812 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:19.725090 5896 server_base.cc:1034] running on GCE node
I20260927 13:43:19.725646 5896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:19.726429 5896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:19.727634 5896 hybrid_clock.cc:648] HybridClock initialized: now 1790516599727591 us; error 52 us; skew 500 ppm
I20260927 13:43:19.727932 5896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:19.730844 5896 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:19.732198 5896 webserver.cc:460] Webserver started at http://127.1.237.195:39733/ using document root <none> and password file <none>
I20260927 13:43:19.732810 5896 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:19.732949 5896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:19.733235 5896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:19.736095 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "952e0e7138524d488cad0b4e52bdb25d"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "39326bcc2081816772ed0a5abb9bb3c5"
server_key_iv: "d4ec1dc1e26b896f26329155206cba6b"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.736922 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "952e0e7138524d488cad0b4e52bdb25d"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "39326bcc2081816772ed0a5abb9bb3c5"
server_key_iv: "d4ec1dc1e26b896f26329155206cba6b"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.741436 5896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:19.744555 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.745715 5896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:19.745879 5896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "952e0e7138524d488cad0b4e52bdb25d"
format_stamp: "Formatted at 2026-09-27 13:43:19 on dist-test-slave-7g3l"
server_key: "39326bcc2081816772ed0a5abb9bb3c5"
server_key_iv: "d4ec1dc1e26b896f26329155206cba6b"
server_key_version: "encryptionkey@0"
I20260927 13:43:19.746059 5896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:19.774825 5896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:19.778792 5896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:19.779064 5896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:19.779932 5896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:19.781100 5896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:19.781200 5896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.781344 5896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:19.781424 5896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:19.813287 5896 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:38755
I20260927 13:43:19.813337 6023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:38755 every 8 connection(s)
I20260927 13:43:19.814531 5896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:19.822728 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 5896
I20260927 13:43:19.822911 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:19.823395 1975 external_mini_cluster.cc:1428] Setting key 131841e60aabab4d58c7207091b199ef
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:19.832656 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.816756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:41427 (local address 127.1.237.254:45859)
0927 13:43:19.816911 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:19.816915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:19.818032 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 13:43:19.819796 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:19.819800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:19.819857 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:19.819935 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:19.821271 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.821938 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.823830 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.824558 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.824685 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.827605 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:19.827611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:19.827615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:19.827651 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:19.829967 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.830623 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.830632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.830634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.830733 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.831114 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.831119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.831122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.831410 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:19.831562 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:19.832149 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 13:43:19.832346 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:43:19.833415 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.817272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45859 (local address 127.1.237.195:41427)
0927 13:43:19.817876 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:19.817916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:19.819567 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:19.820143 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:19.820150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:19.820626 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:19.821145 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:19.821161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.822050 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.822053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:19.823692 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:19.823706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.824773 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.824781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:19.825017 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.825603 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:19.825622 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:19.827437 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:19.830103 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:19.830111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:19.830124 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:19.830482 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:19.830835 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:19.830840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:19.830844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:19.831006 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:19.831539 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:19.831546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:19.831830 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 13:43:19.832061 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 13:43:19.832483 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":237,"threads_started":1}
I20260927 13:43:19.834488 6024 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45859
I20260927 13:43:19.834744 6024 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:19.835417 6024 heartbeater.cc:511] Master 127.1.237.254:45859 requested a full tablet report, sending...
I20260927 13:43:19.836695 5568 ts_manager.cc:194] Registered new tserver with Master: 952e0e7138524d488cad0b4e52bdb25d (127.1.237.195:38755)
I20260927 13:43:19.837535 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:41427
I20260927 13:43:19.837775 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:19.881764 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.868041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.1.237.193:34411)
0927 13:43:19.868544 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:19.868553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:19.868752 (+ 199us) server_negotiation.cc:407] Connection header received
0927 13:43:19.868945 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:19.868958 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:19.869184 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:19.869343 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:19.870530 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.871533 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.872736 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.873290 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.873415 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.876817 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:19.876827 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:19.876838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:19.876867 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:19.879292 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.879882 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.879892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.879900 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.880007 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.880369 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.880375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.880378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.880666 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:19.880960 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:19.881218 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 13:43:19.881453 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
Sep 27 13:43:19 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:19.899519 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.885620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.1.237.254:45859)
0927 13:43:19.885808 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:19.885814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:19.886069 (+ 255us) server_negotiation.cc:407] Connection header received
0927 13:43:19.886219 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:19.886223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:19.886276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:19.886347 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:19.887847 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.888565 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.890366 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.890740 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.890826 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.894839 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:19.894843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:19.894846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:19.894872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:19.897162 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.897760 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.897767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.897771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.897905 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.898228 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.898234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.898237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.898512 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:19.898747 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:19.899031 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 13:43:19.899254 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 13:43:19.899993 5568 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33530
I20260927 13:43:19.900874 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5629
I20260927 13:43:19.912060 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:19.901637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.1.237.193:34411)
0927 13:43:19.901796 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:19.901800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:19.901999 (+ 199us) server_negotiation.cc:407] Connection header received
0927 13:43:19.902112 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:19.902116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:19.902181 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:19.902275 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:19.903434 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.904089 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.905186 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:19.905576 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:19.905661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:19.907855 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:19.907860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:19.907863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:19.907889 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:19.910084 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.910606 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.910610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.910612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.910713 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:19.910988 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:19.910991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:19.910993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:19.911268 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:19.911425 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:19.911677 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 13:43:19.911823 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:43:19.912617 5752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:19.992065 5757 heartbeater.cc:503] Master 127.1.237.254:45859 was elected leader, sending a full tablet report...
I20260927 13:43:19.992654 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5629
W20260927 13:43:20.008496 5554 connection.cc:535] server connection from 127.1.237.193:58773 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:43:20.008837 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5763
Sep 27 13:43:20 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:20.023784 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:20.009634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.1.237.194:36073)
0927 13:43:20.010100 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:20.010106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:20.010129 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:43:20.010181 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:20.010187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:20.010390 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:20.010553 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:20.011775 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:20.012756 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:20.013975 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:20.014476 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:20.014612 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:20.018439 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:20.018450 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:20.018463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:20.018491 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:20.021121 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:20.021790 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:20.021797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:20.021799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:20.021911 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:20.022276 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:20.022283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:20.022286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:20.022600 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:20.022911 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:20.023218 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:43:20.023464 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260927 13:43:20.024976 5886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:20.103547 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5763
I20260927 13:43:20.118563 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5896
Sep 27 13:43:20 dist-test-slave-7g3l krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:20.133342 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:20.119385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.1.237.195:38755)
0927 13:43:20.119929 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:20.119936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:20.119961 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:43:20.120021 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:20.120026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:20.120220 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:20.120376 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:20.121611 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:20.122584 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:20.123901 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:20.124440 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:20.124579 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:20.128526 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:20.128536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:20.128548 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:20.128576 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:20.130982 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:20.131621 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:20.131627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:20.131629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:20.131744 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:20.132054 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:20.132057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:20.132059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:20.132330 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:20.132539 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:20.132795 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:43:20.132961 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":267,"threads_started":1}
I20260927 13:43:20.134549 6019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:20.219549 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5896
I20260927 13:43:20.234738 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 5536
I20260927 13:43:20.235901 5598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:20.332163 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 5536
2026-09-27T13:43:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3609 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:20 dist-test-slave-7g3l krb5kdc[6043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:20 dist-test-slave-7g3l krb5kdc[6043](info): set up 2 sockets
Sep 27 13:43:20 dist-test-slave-7g3l krb5kdc[6043](info): commencing operation
krb5kdc: starting...
W20260927 13:43:21.371402 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:21 dist-test-slave-7g3l krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:21.557940 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46365
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:21.854110 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:21.854383 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:21.854542 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:21.861451 6056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:21.861537 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:21.861614 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:21.861712 6056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:21.861790 6056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:21.870916 6056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:21.872066 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:21.873239 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:21.883819 6062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:21.883857 6061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:21.883886 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:21.884619 6056 server_base.cc:1034] running on GCE node
I20260927 13:43:21.885368 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:21.886616 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:21.887805 6056 hybrid_clock.cc:648] HybridClock initialized: now 1790516601887771 us; error 34 us; skew 500 ppm
I20260927 13:43:21.888144 6056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:21 dist-test-slave-7g3l krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:21.890942 6056 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:21.892122 6056 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:21.892231 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:21.892547 6056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:21.895298 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "f1543a89254b4860967f4a9a1ac83df7"
format_stamp: "Formatted at 2026-09-27 13:43:21 on dist-test-slave-7g3l"
server_key: "c5c8a56162a943341965547f538712bb"
server_key_iv: "d44d238d042bb6111c3782f831dcdd4a"
server_key_version: "encryptionkey@0"
I20260927 13:43:21.896049 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "f1543a89254b4860967f4a9a1ac83df7"
format_stamp: "Formatted at 2026-09-27 13:43:21 on dist-test-slave-7g3l"
server_key: "c5c8a56162a943341965547f538712bb"
server_key_iv: "d44d238d042bb6111c3782f831dcdd4a"
server_key_version: "encryptionkey@0"
I20260927 13:43:21.900543 6056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:21.903791 6070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:21.905011 6056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:21.905166 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "f1543a89254b4860967f4a9a1ac83df7"
format_stamp: "Formatted at 2026-09-27 13:43:21 on dist-test-slave-7g3l"
server_key: "c5c8a56162a943341965547f538712bb"
server_key_iv: "d44d238d042bb6111c3782f831dcdd4a"
server_key_version: "encryptionkey@0"
I20260927 13:43:21.905385 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:21.923365 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:21.926609 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:21.926828 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:21.943133 6056 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46365
I20260927 13:43:21.943192 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46365 every 8 connection(s)
I20260927 13:43:21.943931 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:21.946972 6122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:21.951867 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6056
I20260927 13:43:21.952112 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:21.952863 1975 external_mini_cluster.cc:1428] Setting key efe28f4b4883691e334f7e5579ad3891
I20260927 13:43:21.957425 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: Bootstrap starting.
I20260927 13:43:21.960791 6122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:21.961957 6122 log.cc:824] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:21.964695 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: No bootstrap required, opened a new log
Sep 27 13:43:21 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:21.972110 6122 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } }
I20260927 13:43:21.972438 6122 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:21.972561 6122 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1543a89254b4860967f4a9a1ac83df7, State: Initialized, Role: FOLLOWER
I20260927 13:43:21.973202 6122 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } }
I20260927 13:43:21.973476 6122 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:21.973652 6122 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:21.973829 6122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:21.974691 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:21.954782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.1.237.254:46365)
0927 13:43:21.955606 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:21.955617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:21.955656 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:43:21.957247 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:21.957265 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:21.957639 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:21.957913 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:21.959712 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:21.960971 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:21.962701 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:21.963489 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:21.963604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:21.967884 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:21.967891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:21.967902 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:21.967937 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:21.970786 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:21.971493 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:21.971514 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:21.971520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:21.971659 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:21.972017 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:21.972022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:21.972027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:21.972626 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:21.972908 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:21.973283 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 13:43:21.973653 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":420,"threads_started":1}
I20260927 13:43:21.976140 6122 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } }
I20260927 13:43:21.976578 6122 leader_election.cc:302] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1543a89254b4860967f4a9a1ac83df7; no voters:
I20260927 13:43:21.977164 6122 leader_election.cc:288] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:21.977252 6127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:21.977877 6127 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [term 1 LEADER]: Becoming Leader. State: Replica: f1543a89254b4860967f4a9a1ac83df7, State: Running, Role: LEADER
I20260927 13:43:21.978416 6127 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } }
I20260927 13:43:21.979169 6122 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:21.981396 6129 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1543a89254b4860967f4a9a1ac83df7. Latest consensus state: current_term: 1 leader_uuid: "f1543a89254b4860967f4a9a1ac83df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } } }
I20260927 13:43:21.981496 6128 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1543a89254b4860967f4a9a1ac83df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1543a89254b4860967f4a9a1ac83df7" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46365 } } }
I20260927 13:43:21.981802 6129 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:21.981889 6128 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:21.984308 6134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:21.989154 6134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:21.998565 6134 catalog_manager.cc:1347] Generated new cluster ID: a8ac5fa6be7645fa8f3e8c6f1307cc02
I20260927 13:43:21.998672 6134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:22.029196 6134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:22.030604 6134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:22.042534 6134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: Generated new TSK 0
I20260927 13:43:22.043388 6134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:22.090164 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:22.411921 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:22.412294 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:22.412463 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:22.419333 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:22.419458 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:22.419629 6148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:22.428377 6148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:22.429539 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:22.430670 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:22.442708 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:22.442756 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:22.442989 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:22.443635 6148 server_base.cc:1034] running on GCE node
I20260927 13:43:22.444176 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:22.445061 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:22.446254 6148 hybrid_clock.cc:648] HybridClock initialized: now 1790516602446212 us; error 39 us; skew 500 ppm
I20260927 13:43:22.446558 6148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:22 dist-test-slave-7g3l krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516602, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:22.449397 6148 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:22.450486 6148 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:22.450615 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:22.450906 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:22.453497 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "51aca962d7b54508ab310b5efa1787b5"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "8528e90f6ba99c376593e529de3e93d5"
server_key_iv: "9dd20b8031d0c2c0cb69860c0aa3b497"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.454211 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "51aca962d7b54508ab310b5efa1787b5"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "8528e90f6ba99c376593e529de3e93d5"
server_key_iv: "9dd20b8031d0c2c0cb69860c0aa3b497"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.458606 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 13:43:22.461603 6162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:22.462697 6148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:43:22.462881 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "51aca962d7b54508ab310b5efa1787b5"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "8528e90f6ba99c376593e529de3e93d5"
server_key_iv: "9dd20b8031d0c2c0cb69860c0aa3b497"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.463227 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:22.477831 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:22.481527 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:22.481763 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:22.482537 6148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:22.483666 6148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:22.483740 6148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:22.483872 6148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:22.483939 6148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:22.513279 6148 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:44501
I20260927 13:43:22.513309 6274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:44501 every 8 connection(s)
I20260927 13:43:22.514047 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:22.516235 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6148
I20260927 13:43:22.516454 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:22.517174 1975 external_mini_cluster.cc:1428] Setting key af02c3254183b61d4fb9cf03f414b9ff
Sep 27 13:43:22 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516602, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:22.535882 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:22.516334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:53631 (local address 127.1.237.254:46365)
0927 13:43:22.516799 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:22.516807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:22.517800 (+ 993us) server_negotiation.cc:407] Connection header received
0927 13:43:22.519816 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:22.519820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:22.519882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:22.519964 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:22.521936 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:22.522740 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:22.525175 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:22.525555 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:22.525684 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:22.529154 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:22.529163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:22.529170 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:22.529211 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:22.532752 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:22.533504 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:22.533512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:22.533527 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:22.533667 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:22.534059 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:22.534066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:22.534071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:22.534448 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:22.534628 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:22.535244 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 13:43:22.535453 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260927 13:43:22.536491 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:22.516872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46365 (local address 127.1.237.193:53631)
0927 13:43:22.517621 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:22.517666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:22.519544 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:22.520218 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:22.520231 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:22.521023 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:22.521778 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:22.521807 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:22.522889 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:22.522897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:22.525027 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:22.525038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:22.525758 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:22.525767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:22.525985 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:22.526624 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:22.526644 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:22.528947 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:22.532893 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:22.532903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:22.532919 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:22.533372 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:22.533782 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:22.533785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:22.533799 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:22.533943 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:22.534592 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:22.534598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:22.534871 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 13:43:22.535075 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 13:43:22.535466 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:43:22.537945 6275 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46365
I20260927 13:43:22.538272 6275 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:22.538880 6275 heartbeater.cc:511] Master 127.1.237.254:46365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:43:22.541141 6087 ts_manager.cc:194] Registered new tserver with Master: 51aca962d7b54508ab310b5efa1787b5 (127.1.237.193:44501)
I20260927 13:43:22.543435 6087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:53631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:22.559803 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:22.853689 6281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:22.853935 6281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:22.854095 6281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:22.861590 6281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:22.861685 6281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:22.861886 6281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:22.870971 6281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:22.872291 6281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:22.873562 6281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:22.886368 6286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:22.886318 6289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:22.886344 6287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:22.887048 6281 server_base.cc:1034] running on GCE node
I20260927 13:43:22.887684 6281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:22.888609 6281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:22.889828 6281 hybrid_clock.cc:648] HybridClock initialized: now 1790516602889789 us; error 54 us; skew 500 ppm
I20260927 13:43:22.890131 6281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:22 dist-test-slave-7g3l krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516602, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:22.893134 6281 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:22.894222 6281 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:22.894346 6281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:22.894642 6281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:22.897419 6281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "44ebfae304db46df91a1e445ab2f6f7c"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "88f911602ab538afabe0f969f5b57c4f"
server_key_iv: "b621f5d07ec8dadcde4ab79897923db4"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.898195 6281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "44ebfae304db46df91a1e445ab2f6f7c"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "88f911602ab538afabe0f969f5b57c4f"
server_key_iv: "b621f5d07ec8dadcde4ab79897923db4"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.902884 6281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 13:43:22.906167 6295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:22.907480 6281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:22.907653 6281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "44ebfae304db46df91a1e445ab2f6f7c"
format_stamp: "Formatted at 2026-09-27 13:43:22 on dist-test-slave-7g3l"
server_key: "88f911602ab538afabe0f969f5b57c4f"
server_key_iv: "b621f5d07ec8dadcde4ab79897923db4"
server_key_version: "encryptionkey@0"
I20260927 13:43:22.907859 6281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:22.942584 6281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:22.946951 6281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:22.947350 6281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:22.955513 6281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:22.959872 6281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:22.959998 6281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:22.960142 6281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:22.960232 6281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:23.003844 6281 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:45013
I20260927 13:43:23.003880 6407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:45013 every 8 connection(s)
I20260927 13:43:23.004690 6281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:23.007992 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6281
I20260927 13:43:23.008224 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:23.008812 1975 external_mini_cluster.cc:1428] Setting key a2d33b4a009f128581cad343df9f5665
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516602, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:23.026425 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.007224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:54001 (local address 127.1.237.254:46365)
0927 13:43:23.007471 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.007480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.008948 (+ 1468us) server_negotiation.cc:407] Connection header received
0927 13:43:23.011127 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.011132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.011232 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.011353 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.012991 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.013806 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.015333 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.015768 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.015895 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.019302 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.019309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.019316 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.019358 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.022962 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.023849 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.023856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.023860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.024005 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.024525 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.024532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.024536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.024897 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.025065 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.025877 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.026085 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
I20260927 13:43:23.027762 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.007817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46365 (local address 127.1.237.194:54001)
0927 13:43:23.008756 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.008803 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.010836 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.011608 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.011620 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.012224 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:23.012838 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.012856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.013953 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.013960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.015143 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.015209 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.015969 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.015977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.016194 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.016818 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:23.016836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:23.019046 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:23.023095 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:23.023105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:23.023123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:23.023729 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:23.024133 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:23.024138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:23.024143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:23.024371 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:23.025034 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:23.025043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:23.025431 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.025745 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.026212 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":403,"threads_started":1}
I20260927 13:43:23.029237 6408 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46365
I20260927 13:43:23.029532 6408 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:23.030210 6408 heartbeater.cc:511] Master 127.1.237.254:46365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
I20260927 13:43:23.031631 6087 ts_manager.cc:194] Registered new tserver with Master: 44ebfae304db46df91a1e445ab2f6f7c (127.1.237.194:45013)
I20260927 13:43:23.032401 6087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:54001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:23.052178 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:23.351879 6413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:23.352130 6413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:23.352281 6413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:23.359093 6413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:23.359236 6413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:23.359419 6413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:23.368773 6413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:23.369879 6413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:23.371065 6413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:23.383327 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:23.383286 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:23.383370 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:23.388525 6413 server_base.cc:1034] running on GCE node
I20260927 13:43:23.389067 6413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:23.389820 6413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:23.391038 6413 hybrid_clock.cc:648] HybridClock initialized: now 1790516603390995 us; error 35 us; skew 500 ppm
I20260927 13:43:23.391374 6413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:23.394176 6413 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:23.395336 6413 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:23.395452 6413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:23.395763 6413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:23.398391 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "d58f0b266da0414887f0a793e30ae803"
format_stamp: "Formatted at 2026-09-27 13:43:23 on dist-test-slave-7g3l"
server_key: "53a762d31f8e0c92833373c182162e43"
server_key_iv: "4ce54e4e9302918bad8e79d81e4d2e10"
server_key_version: "encryptionkey@0"
I20260927 13:43:23.399286 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "d58f0b266da0414887f0a793e30ae803"
format_stamp: "Formatted at 2026-09-27 13:43:23 on dist-test-slave-7g3l"
server_key: "53a762d31f8e0c92833373c182162e43"
server_key_iv: "4ce54e4e9302918bad8e79d81e4d2e10"
server_key_version: "encryptionkey@0"
I20260927 13:43:23.403841 6413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:23.407222 6427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:23.408648 6413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:43:23.408797 6413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "d58f0b266da0414887f0a793e30ae803"
format_stamp: "Formatted at 2026-09-27 13:43:23 on dist-test-slave-7g3l"
server_key: "53a762d31f8e0c92833373c182162e43"
server_key_iv: "4ce54e4e9302918bad8e79d81e4d2e10"
server_key_version: "encryptionkey@0"
I20260927 13:43:23.409001 6413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:23.427482 6413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:23.431389 6413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:23.431630 6413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:23.432440 6413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:23.433619 6413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:23.433735 6413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:23.433862 6413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:23.433981 6413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:23.468707 6413 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:45377
I20260927 13:43:23.468756 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:45377 every 8 connection(s)
I20260927 13:43:23.469491 6413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:23.476357 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6413
I20260927 13:43:23.476545 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:23.476919 1975 external_mini_cluster.cc:1428] Setting key 798d48f935a426b8a91959eba83c0469
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:23.487579 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.471744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:47145 (local address 127.1.237.254:46365)
0927 13:43:23.471912 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.471916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.472964 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 13:43:23.474643 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.474648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.474730 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.474830 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.476278 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.477159 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.479035 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.479639 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.479763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.483049 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.483053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.483056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.483084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.485300 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.485887 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.485890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.485892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.485991 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.486301 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.486304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.486306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.486547 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.486651 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.487203 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.487347 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:43:23.488438 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.472183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46365 (local address 127.1.237.195:47145)
0927 13:43:23.472815 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.472850 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.474380 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.475060 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.475067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.475584 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:23.476121 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.476137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.477308 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.477312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.478878 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.478891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.479837 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.479848 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.480152 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.480697 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:23.480717 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:23.482855 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:23.485411 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:23.485416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:23.485426 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:23.485797 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:23.486075 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:23.486078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:23.486080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:23.486220 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:23.486648 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:23.486653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:23.486894 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.487094 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.487463 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":253,"threads_started":1}
I20260927 13:43:23.489630 6540 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46365
I20260927 13:43:23.489854 6540 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:23.490456 6540 heartbeater.cc:511] Master 127.1.237.254:46365 requested a full tablet report, sending...
I20260927 13:43:23.491909 6087 ts_manager.cc:194] Registered new tserver with Master: d58f0b266da0414887f0a793e30ae803 (127.1.237.195:45377)
I20260927 13:43:23.492673 6087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:47145
I20260927 13:43:23.503304 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:43:23.519902 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.507683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.1.237.254:46365)
0927 13:43:23.507896 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.507904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.508358 (+ 454us) server_negotiation.cc:407] Connection header received
0927 13:43:23.508536 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.508541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.508602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.508700 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.510106 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.510846 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.512186 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.512696 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.512831 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.515257 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.515261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.515264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.515291 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.517554 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.518093 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.518100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.518104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.518214 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.518583 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.518590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.518594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.518851 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.519099 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.519426 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.519615 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:43:23.523144 6087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:23.529537 6087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:23.546164 6275 heartbeater.cc:503] Master 127.1.237.254:46365 was elected leader, sending a full tablet report...
I20260927 13:43:23.550405 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.542484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.1.237.195:45377)
0927 13:43:23.543653 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.543658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.543704 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:43:23.543788 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.543794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.543991 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.544172 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:23.544676 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545778 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.548132 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.549345 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.549469 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.549686 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.549936 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.550088 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":906,"threads_started":1}
I20260927 13:43:23.550489 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.542122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:45377 (local address 127.0.0.1:58804)
0927 13:43:23.542605 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.542621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.542725 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.544160 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.544164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.544186 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:23.544492 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.544500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545917 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.545936 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.547987 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.547997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.549535 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.549541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.549716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.549760 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.549887 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.550073 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"thread_start_us":690,"threads_started":1}
I20260927 13:43:23.551192 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.540183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:45013 (local address 127.0.0.1:48582)
0927 13:43:23.541714 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.541753 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.541938 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.542830 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.542837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.542882 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:23.543828 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.543841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545561 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.545564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.548454 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.548466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.550365 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.550371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.550554 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.550581 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.550661 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.550781 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":132,"spinlock_wait_cycles":4736,"thread_start_us":1374,"threads_started":1}
I20260927 13:43:23.551366 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.540820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.1.237.193:44501)
0927 13:43:23.542320 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.542327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.542349 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:43:23.542429 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.542437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.542646 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.542950 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:23.543769 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545355 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.548791 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.549862 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.550600 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.550813 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.550879 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.551041 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1922,"threads_started":1}
I20260927 13:43:23.551615 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.540237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44501 (local address 127.0.0.1:37786)
0927 13:43:23.541789 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.541806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.541936 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.542830 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.542837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.542880 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:23.543297 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.543310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545489 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.545494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.548665 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.548674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.550078 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.550083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.550252 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.550282 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.550353 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.550483 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":162,"spinlock_wait_cycles":7808,"thread_start_us":1193,"threads_started":1}
I20260927 13:43:23.551856 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.540277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.1.237.194:45013)
0927 13:43:23.541378 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.541389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.541842 (+ 453us) server_negotiation.cc:407] Connection header received
0927 13:43:23.542172 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.542179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.542403 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.542556 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:23.544000 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.545401 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.548605 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.549914 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.550046 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.550220 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.551428 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.551570 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":806,"threads_started":1}
I20260927 13:43:23.554085 6475 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2881b020954a768dae8036926a4688 (DEFAULT_TABLE table=test-table [id=3c41edb4154048ceaac3e06282693750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:23.555111 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2881b020954a768dae8036926a4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:23.554925 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2881b020954a768dae8036926a4688 (DEFAULT_TABLE table=test-table [id=3c41edb4154048ceaac3e06282693750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:23.554925 6210 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2881b020954a768dae8036926a4688 (DEFAULT_TABLE table=test-table [id=3c41edb4154048ceaac3e06282693750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:23.556186 6210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2881b020954a768dae8036926a4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:23.556186 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2881b020954a768dae8036926a4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:23.565363 6558 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Bootstrap starting.
I20260927 13:43:23.568266 6558 tablet_bootstrap.cc:654] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:23.569064 6559 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Bootstrap starting.
I20260927 13:43:23.569782 6560 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Bootstrap starting.
I20260927 13:43:23.570437 6558 log.cc:824] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:23.572829 6558 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: No bootstrap required, opened a new log
I20260927 13:43:23.573024 6560 tablet_bootstrap.cc:654] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:23.573038 6559 tablet_bootstrap.cc:654] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:23.573081 6558 ts_tablet_manager.cc:1397] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260927 13:43:23.574136 6559 log.cc:824] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:23.574137 6560 log.cc:824] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:23.577772 6560 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: No bootstrap required, opened a new log
I20260927 13:43:23.578013 6560 ts_tablet_manager.cc:1397] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 13:43:23.578270 6559 tablet_bootstrap.cc:492] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: No bootstrap required, opened a new log
I20260927 13:43:23.578500 6559 ts_tablet_manager.cc:1397] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 13:43:23.582437 6558 raft_consensus.cc:348] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.582796 6558 raft_consensus.cc:374] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:23.582885 6558 raft_consensus.cc:729] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58f0b266da0414887f0a793e30ae803, State: Initialized, Role: FOLLOWER
I20260927 13:43:23.583451 6558 consensus_queue.cc:260] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.584384 6540 heartbeater.cc:503] Master 127.1.237.254:46365 was elected leader, sending a full tablet report...
I20260927 13:43:23.584900 6558 ts_tablet_manager.cc:1428] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 13:43:23.584825 6560 raft_consensus.cc:348] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.585201 6560 raft_consensus.cc:374] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:23.585275 6560 raft_consensus.cc:729] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ebfae304db46df91a1e445ab2f6f7c, State: Initialized, Role: FOLLOWER
I20260927 13:43:23.585767 6560 consensus_queue.cc:260] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.586783 6408 heartbeater.cc:503] Master 127.1.237.254:46365 was elected leader, sending a full tablet report...
I20260927 13:43:23.587440 6560 ts_tablet_manager.cc:1428] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20260927 13:43:23.587615 6559 raft_consensus.cc:348] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.588112 6559 raft_consensus.cc:374] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:23.588207 6559 raft_consensus.cc:729] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51aca962d7b54508ab310b5efa1787b5, State: Initialized, Role: FOLLOWER
I20260927 13:43:23.588750 6559 consensus_queue.cc:260] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.590432 6559 ts_tablet_manager.cc:1428] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 13:43:23.610145 6564 raft_consensus.cc:482] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:23.610313 6564 raft_consensus.cc:504] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.611747 6564 leader_election.cc:288] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ebfae304db46df91a1e445ab2f6f7c (127.1.237.194:45013), 51aca962d7b54508ab310b5efa1787b5 (127.1.237.193:44501)
I20260927 13:43:23.617523 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.612196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:42987 (local address 127.1.237.194:45013)
0927 13:43:23.612462 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.612470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.612491 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:43:23.612563 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.612571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.612673 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.612770 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:23.613162 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.613939 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.615788 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.616811 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.616931 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.616985 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.617292 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.617366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 13:43:23.617646 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.611886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:45013 (local address 127.1.237.195:42987)
0927 13:43:23.612180 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.612200 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.612327 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.612817 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.612825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.612850 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:23.613038 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.613048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.614074 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.614081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.615623 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.615655 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.616976 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.616981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.617147 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.617177 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.617270 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.617386 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":1}
I20260927 13:43:23.618461 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.612323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:34175 (local address 127.1.237.193:44501)
0927 13:43:23.612529 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.612537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.613046 (+ 509us) server_negotiation.cc:407] Connection header received
0927 13:43:23.613277 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.613281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.613329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.613420 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:23.613854 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.614611 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.616668 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.617721 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.617907 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.618006 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.618205 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.618314 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:43:23.618587 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.612272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:44501 (local address 127.1.237.195:34175)
0927 13:43:23.612939 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:23.612954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:23.613094 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:23.613431 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:23.613435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:23.613457 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:23.613723 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.613731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.614746 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.614751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.616515 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:23.616527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.617936 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.617944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:23.618067 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.618089 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:43:23.618155 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 13:43:23.618265 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260927 13:43:23.618630 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2881b020954a768dae8036926a4688" candidate_uuid: "d58f0b266da0414887f0a793e30ae803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ebfae304db46df91a1e445ab2f6f7c" is_pre_election: true
I20260927 13:43:23.619087 6363 raft_consensus.cc:2454] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58f0b266da0414887f0a793e30ae803 in term 0.
I20260927 13:43:23.619676 6431 leader_election.cc:302] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ebfae304db46df91a1e445ab2f6f7c, d58f0b266da0414887f0a793e30ae803; no voters:
I20260927 13:43:23.619644 6230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2881b020954a768dae8036926a4688" candidate_uuid: "d58f0b266da0414887f0a793e30ae803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aca962d7b54508ab310b5efa1787b5" is_pre_election: true
I20260927 13:43:23.620024 6564 raft_consensus.cc:2781] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:23.620169 6564 raft_consensus.cc:482] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:23.620107 6230 raft_consensus.cc:2454] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58f0b266da0414887f0a793e30ae803 in term 0.
I20260927 13:43:23.620297 6564 raft_consensus.cc:3037] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:23.622324 6564 raft_consensus.cc:504] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.622927 6564 leader_election.cc:288] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [CANDIDATE]: Term 1 election: Requested vote from peers 44ebfae304db46df91a1e445ab2f6f7c (127.1.237.194:45013), 51aca962d7b54508ab310b5efa1787b5 (127.1.237.193:44501)
I20260927 13:43:23.623255 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2881b020954a768dae8036926a4688" candidate_uuid: "d58f0b266da0414887f0a793e30ae803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ebfae304db46df91a1e445ab2f6f7c"
I20260927 13:43:23.623353 6230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2881b020954a768dae8036926a4688" candidate_uuid: "d58f0b266da0414887f0a793e30ae803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aca962d7b54508ab310b5efa1787b5"
I20260927 13:43:23.623415 6363 raft_consensus.cc:3037] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:23.623548 6230 raft_consensus.cc:3037] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:23.625394 6363 raft_consensus.cc:2454] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58f0b266da0414887f0a793e30ae803 in term 1.
I20260927 13:43:23.625466 6230 raft_consensus.cc:2454] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58f0b266da0414887f0a793e30ae803 in term 1.
I20260927 13:43:23.625823 6431 leader_election.cc:302] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44ebfae304db46df91a1e445ab2f6f7c, 51aca962d7b54508ab310b5efa1787b5, d58f0b266da0414887f0a793e30ae803; no voters:
I20260927 13:43:23.626092 6564 raft_consensus.cc:2781] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:23.626454 6564 raft_consensus.cc:686] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 1 LEADER]: Becoming Leader. State: Replica: d58f0b266da0414887f0a793e30ae803, State: Running, Role: LEADER
I20260927 13:43:23.626852 6564 consensus_queue.cc:237] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } }
I20260927 13:43:23.630779 6086 catalog_manager.cc:5495] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 reported cstate change: term changed from 0 to 1, leader changed from <none> to d58f0b266da0414887f0a793e30ae803 (127.1.237.195). New cstate: current_term: 1 leader_uuid: "d58f0b266da0414887f0a793e30ae803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d58f0b266da0414887f0a793e30ae803" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45377 } health_report { overall_health: HEALTHY } } }
I20260927 13:43:23.645402 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.640617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.1.237.195:45377)
0927 13:43:23.640781 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.640786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.640832 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:43:23.641056 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.641059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.641140 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.641276 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:23.641622 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.642262 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.643576 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.643972 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.644093 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.644519 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:23.644911 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:23.645083 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.645124 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.645189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:23.651980 6230 raft_consensus.cc:1261] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 1 FOLLOWER]: Refusing update from remote peer d58f0b266da0414887f0a793e30ae803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:23.651978 6363 raft_consensus.cc:1261] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 1 FOLLOWER]: Refusing update from remote peer d58f0b266da0414887f0a793e30ae803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:23.652830 6568 consensus_queue.cc:1035] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51aca962d7b54508ab310b5efa1787b5" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:23.653158 6564 consensus_queue.cc:1035] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ebfae304db46df91a1e445ab2f6f7c" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:23.681949 6087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:23.682222 6087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:23.685199 6087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1543a89254b4860967f4a9a1ac83df7: Sending DeleteTablet for 3 replicas of tablet 5b2881b020954a768dae8036926a4688
I20260927 13:43:23.686136 6343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2881b020954a768dae8036926a4688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:23 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:48582
I20260927 13:43:23.686241 6475 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2881b020954a768dae8036926a4688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:23 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58804
I20260927 13:43:23.686818 6210 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2881b020954a768dae8036926a4688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:23 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:37786
I20260927 13:43:23.686939 6581 tablet_replica.cc:330] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: stopping tablet replica
I20260927 13:43:23.687307 6581 raft_consensus.cc:2229] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:23.687527 6583 tablet_replica.cc:330] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: stopping tablet replica
I20260927 13:43:23.687814 6581 raft_consensus.cc:2258] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:23.687922 6583 raft_consensus.cc:2229] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:23.688962 6583 raft_consensus.cc:2258] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:23.689538 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6148
I20260927 13:43:23.690492 6581 ts_tablet_manager.cc:1905] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:23.700016 6582 tablet_replica.cc:330] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: stopping tablet replica
I20260927 13:43:23.700424 6583 ts_tablet_manager.cc:1905] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:23.700603 6582 raft_consensus.cc:2229] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:23.701290 6582 raft_consensus.cc:2258] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:23.703125 6582 ts_tablet_manager.cc:1905] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:23.708225 6583 ts_tablet_manager.cc:1918] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:23.708559 6583 log.cc:1196] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/5b2881b020954a768dae8036926a4688
I20260927 13:43:23.709218 6583 ts_tablet_manager.cc:1939] T 5b2881b020954a768dae8036926a4688 P 51aca962d7b54508ab310b5efa1787b5: Deleting consensus metadata
I20260927 13:43:23.709825 6581 ts_tablet_manager.cc:1918] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:23.709993 6581 log.cc:1196] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/5b2881b020954a768dae8036926a4688
I20260927 13:43:23.710596 6581 ts_tablet_manager.cc:1939] T 5b2881b020954a768dae8036926a4688 P 44ebfae304db46df91a1e445ab2f6f7c: Deleting consensus metadata
I20260927 13:43:23.711367 6072 catalog_manager.cc:4841] TS 51aca962d7b54508ab310b5efa1787b5 (127.1.237.193:44501): tablet 5b2881b020954a768dae8036926a4688 (table test-table [id=3c41edb4154048ceaac3e06282693750]) successfully deleted
I20260927 13:43:23.712375 6074 catalog_manager.cc:4841] TS 44ebfae304db46df91a1e445ab2f6f7c (127.1.237.194:45013): tablet 5b2881b020954a768dae8036926a4688 (table test-table [id=3c41edb4154048ceaac3e06282693750]) successfully deleted
I20260927 13:43:23.712430 6582 ts_tablet_manager.cc:1918] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:23.712581 6582 log.cc:1196] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/5b2881b020954a768dae8036926a4688
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:23.713104 6582 ts_tablet_manager.cc:1939] T 5b2881b020954a768dae8036926a4688 P d58f0b266da0414887f0a793e30ae803: Deleting consensus metadata
I20260927 13:43:23.714447 6074 catalog_manager.cc:4841] TS d58f0b266da0414887f0a793e30ae803 (127.1.237.195:45377): tablet 5b2881b020954a768dae8036926a4688 (table test-table [id=3c41edb4154048ceaac3e06282693750]) successfully deleted
I20260927 13:43:23.720121 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.691889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.1.237.193:44501)
0927 13:43:23.692077 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.692084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.702587 (+ 10503us) server_negotiation.cc:407] Connection header received
0927 13:43:23.702954 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.702959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.703041 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.703111 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.704799 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.705993 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.707880 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.708629 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.708731 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.713910 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.713922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.713939 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.713991 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.717526 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.718112 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.718120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.718124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.718266 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.718579 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.718584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.718587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.719054 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.719386 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.719568 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.719773 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:43:23.720768 6270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:23.803874 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6148
I20260927 13:43:23.821506 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6281
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:23.836817 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.822306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.1.237.194:45013)
0927 13:43:23.822442 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.822446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.822468 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:43:23.822672 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.822676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.822737 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.822827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.824375 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.825087 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.826687 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.827018 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.827092 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.831507 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.831519 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.831535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.831570 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.834204 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.834812 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.834818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.834820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.834930 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.835423 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.835431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.835435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.835762 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.836038 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.836339 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.836541 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 13:43:23.837460 6403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:23.922453 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6281
I20260927 13:43:23.943929 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6413
Sep 27 13:43:23 dist-test-slave-7g3l krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:23.963296 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:23.944654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.1.237.195:45377)
0927 13:43:23.944817 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:23.944823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:23.945152 (+ 329us) server_negotiation.cc:407] Connection header received
0927 13:43:23.945319 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:23.945325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:23.945408 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:23.945526 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:23.947098 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.948297 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.950024 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:23.950681 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:23.950831 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:23.956305 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:23.956322 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:23.956340 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:23.956384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:23.960253 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.961046 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.961055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.961059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.961252 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:23.961681 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:23.961686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:23.961690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:23.962106 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:23.962397 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:23.962696 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 13:43:23.962891 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:43:23.963990 6535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:24.078365 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6413
I20260927 13:43:24.096791 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6056
I20260927 13:43:24.097838 6117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:24.201282 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6056
2026-09-27T13:43:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3872 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T13:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:24Z Disabled control of system clock
I20260927 13:43:24.262815 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36015
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:43:24.560729 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:24.560973 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:24.561050 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:24.567807 6592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:24.567893 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:24.567929 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:24.567981 6592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:24.568022 6592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:24.576634 6592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:24.577679 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:24.579219 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:24.590359 6597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:24.590390 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:24.590415 6598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:24.591055 6592 server_base.cc:1034] running on GCE node
I20260927 13:43:24.591774 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:24.592937 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:24.594123 6592 hybrid_clock.cc:648] HybridClock initialized: now 1790516604594078 us; error 46 us; skew 500 ppm
I20260927 13:43:24.594437 6592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:24.596478 6592 webserver.cc:460] Webserver started at http://127.1.237.254:44233/ using document root <none> and password file <none>
I20260927 13:43:24.597111 6592 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:24.597234 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:24.597581 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:24.600214 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "1706f2f4524b4d6eb40a1ab07a644aa8"
format_stamp: "Formatted at 2026-09-27 13:43:24 on dist-test-slave-7g3l"
server_key: "44234268527f5e497e3204d09f1878c6"
server_key_iv: "a5279748bc3f24ba1f1c02e0000eee11"
server_key_version: "encryptionkey@0"
I20260927 13:43:24.600958 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "1706f2f4524b4d6eb40a1ab07a644aa8"
format_stamp: "Formatted at 2026-09-27 13:43:24 on dist-test-slave-7g3l"
server_key: "44234268527f5e497e3204d09f1878c6"
server_key_iv: "a5279748bc3f24ba1f1c02e0000eee11"
server_key_version: "encryptionkey@0"
I20260927 13:43:24.605453 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:24.608479 6606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:24.609648 6592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:24.609799 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "1706f2f4524b4d6eb40a1ab07a644aa8"
format_stamp: "Formatted at 2026-09-27 13:43:24 on dist-test-slave-7g3l"
server_key: "44234268527f5e497e3204d09f1878c6"
server_key_iv: "a5279748bc3f24ba1f1c02e0000eee11"
server_key_version: "encryptionkey@0"
I20260927 13:43:24.610008 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:24.615232 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:24.615885 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:24.616147 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:24.633044 6592 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36015
I20260927 13:43:24.633081 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36015 every 8 connection(s)
I20260927 13:43:24.634271 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:24.636700 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6592
I20260927 13:43:24.636879 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:24.637468 1975 external_mini_cluster.cc:1428] Setting key 6e0968427855746354182efab53252ec
I20260927 13:43:24.637943 6658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:24.645490 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:24.639372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.1.237.254:36015)
0927 13:43:24.640322 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:24.640335 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:24.640375 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:43:24.642241 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:24.642261 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:24.642645 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:24.643017 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:24.644027 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:24.644032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:24.644043 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:24.644054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:24.644105 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:24.644122 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:24.644349 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:24.644497 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 13:43:24.644717 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
I20260927 13:43:24.649868 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: Bootstrap starting.
I20260927 13:43:24.652889 6658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:24.653797 6658 log.cc:824] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:24.656400 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: No bootstrap required, opened a new log
I20260927 13:43:24.663611 6658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } }
I20260927 13:43:24.663980 6658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:24.664104 6658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1706f2f4524b4d6eb40a1ab07a644aa8, State: Initialized, Role: FOLLOWER
I20260927 13:43:24.664623 6658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } }
I20260927 13:43:24.664846 6658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:24.664995 6658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:24.665159 6658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:24.667203 6658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } }
I20260927 13:43:24.667682 6658 leader_election.cc:302] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1706f2f4524b4d6eb40a1ab07a644aa8; no voters:
I20260927 13:43:24.668234 6658 leader_election.cc:288] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:24.668354 6663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:24.669054 6663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 1706f2f4524b4d6eb40a1ab07a644aa8, State: Running, Role: LEADER
I20260927 13:43:24.669611 6663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } }
I20260927 13:43:24.670372 6658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:24.672273 6665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1706f2f4524b4d6eb40a1ab07a644aa8. Latest consensus state: current_term: 1 leader_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } } }
I20260927 13:43:24.672609 6665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:24.672463 6664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1706f2f4524b4d6eb40a1ab07a644aa8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36015 } } }
I20260927 13:43:24.672842 6664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:24.675515 6669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:24.680687 6669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:24.688231 6669 catalog_manager.cc:1347] Generated new cluster ID: 5658dc04a5774db7ab03f97d4a23a8de
I20260927 13:43:24.688339 6669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:24.701354 6669 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:24.701817 6669 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:24.718616 6669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: Generated new TSK 0
I20260927 13:43:24.719756 6669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:43:24.731482 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--builtin_ntp_servers=127.1.237.212:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 13:43:25.040731 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:25.040987 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:25.041177 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:25.048221 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:25.048370 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:25.048588 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:25.057639 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:25.058789 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:25.060454 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:25.072818 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.072876 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.072898 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:25.073638 6682 server_base.cc:1034] running on GCE node
I20260927 13:43:25.074164 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:25.074990 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:25.076186 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790516605076135 us; error 53 us; skew 500 ppm
I20260927 13:43:25.076474 6682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:25.078452 6682 webserver.cc:460] Webserver started at http://127.1.237.193:37591/ using document root <none> and password file <none>
I20260927 13:43:25.079069 6682 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:25.079216 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:25.079522 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:25.082123 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "9c873aaf559443959c6ccf0ff025d296"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "20310761f5a77324e9185c19d438b78c"
server_key_iv: "c83d609f5404cac71c51a81f4ae68cd9"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.082846 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "9c873aaf559443959c6ccf0ff025d296"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "20310761f5a77324e9185c19d438b78c"
server_key_iv: "c83d609f5404cac71c51a81f4ae68cd9"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.087543 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:43:25.090619 6696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.092105 6682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:25.092293 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "9c873aaf559443959c6ccf0ff025d296"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "20310761f5a77324e9185c19d438b78c"
server_key_iv: "c83d609f5404cac71c51a81f4ae68cd9"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.092526 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:25.097836 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:25.098543 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:25.098819 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:25.099843 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:25.101143 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:25.101267 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.101398 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:25.101485 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.132644 6682 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:42661
I20260927 13:43:25.132689 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:42661 every 8 connection(s)
I20260927 13:43:25.133805 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:25.138514 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6682
I20260927 13:43:25.138700 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:25.139338 1975 external_mini_cluster.cc:1428] Setting key 0a1b2d4bdf8d590ec3327633fe129da6
I20260927 13:43:25.141296 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.136100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:35719 (local address 127.1.237.254:36015)
0927 13:43:25.136253 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:25.136257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:25.137428 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 13:43:25.139065 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:25.139071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:25.139142 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:25.139298 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:25.140619 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:25.140624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:25.140627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:25.140641 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:25.140671 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:25.140681 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:25.140772 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:25.140991 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 13:43:25.141065 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:25.142132 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--builtin_ntp_servers=127.1.237.212:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 13:43:25.142287 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.136471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36015 (local address 127.1.237.193:35719)
0927 13:43:25.137292 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:25.137323 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:25.138809 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:25.139509 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:25.139516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:25.139603 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:25.140307 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:25.140311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:25.140358 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:25.140454 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:25.140778 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:25.140787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:25.140794 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:43:25.141105 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 13:43:25.141345 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":250,"threads_started":1}
I20260927 13:43:25.143627 6809 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36015
I20260927 13:43:25.143884 6809 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:25.144405 6809 heartbeater.cc:511] Master 127.1.237.254:36015 requested a full tablet report, sending...
I20260927 13:43:25.146565 6623 ts_manager.cc:194] Registered new tserver with Master: 9c873aaf559443959c6ccf0ff025d296 (127.1.237.193:42661)
W20260927 13:43:25.458604 6812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:25.458920 6812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:25.459077 6812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:25.465934 6812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:25.466085 6812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:25.466252 6812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:25.474887 6812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:25.476238 6812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:25.477910 6812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:25.489960 6818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.489977 6820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.490010 6817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:25.490849 6812 server_base.cc:1034] running on GCE node
I20260927 13:43:25.491432 6812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:25.492197 6812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:25.493419 6812 hybrid_clock.cc:648] HybridClock initialized: now 1790516605493376 us; error 55 us; skew 500 ppm
I20260927 13:43:25.493711 6812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:25.495823 6812 webserver.cc:460] Webserver started at http://127.1.237.194:34947/ using document root <none> and password file <none>
I20260927 13:43:25.496477 6812 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:25.496601 6812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:25.496896 6812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:25.499554 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "47911ec8798141229a2269f150760a18"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "9b02c96bd29ff259ad21471ed3430941"
server_key_iv: "f815118bed1120b5e752aabcbe7592e5"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.500437 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "47911ec8798141229a2269f150760a18"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "9b02c96bd29ff259ad21471ed3430941"
server_key_iv: "f815118bed1120b5e752aabcbe7592e5"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.504894 6812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 13:43:25.507887 6826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.509087 6812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:25.509253 6812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "47911ec8798141229a2269f150760a18"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "9b02c96bd29ff259ad21471ed3430941"
server_key_iv: "f815118bed1120b5e752aabcbe7592e5"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.509465 6812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:25.514757 6812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:25.515537 6812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:25.515782 6812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:25.516583 6812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:25.517781 6812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:25.517908 6812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.518030 6812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:25.518114 6812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.549147 6812 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:44339
I20260927 13:43:25.549156 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:44339 every 8 connection(s)
I20260927 13:43:25.550710 6812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:25.559954 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.554012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52313 (local address 127.1.237.254:36015)
0927 13:43:25.554310 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:25.554317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:25.555821 (+ 1504us) server_negotiation.cc:407] Connection header received
0927 13:43:25.557935 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:25.557940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:25.558009 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:25.558127 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:25.559251 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:25.559256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:25.559261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:25.559279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:25.559317 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:25.559329 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:25.559445 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:25.559633 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 13:43:25.559696 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 13:43:25.560916 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.554497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36015 (local address 127.1.237.194:52313)
0927 13:43:25.555653 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:25.555697 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:25.557672 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:25.558331 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:25.558339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:25.558418 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:25.558949 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:25.558951 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:25.558985 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:25.559056 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:25.559418 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:25.559427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:25.559434 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:43:25.559707 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 13:43:25.559926 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":330,"threads_started":1}
I20260927 13:43:25.560937 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6812
I20260927 13:43:25.561110 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:25.561553 1975 external_mini_cluster.cc:1428] Setting key b128e341f8b5d873870b6d34f969236b
I20260927 13:43:25.562322 6939 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36015
I20260927 13:43:25.562567 6939 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:25.563110 6939 heartbeater.cc:511] Master 127.1.237.254:36015 requested a full tablet report, sending...
I20260927 13:43:25.564221 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--builtin_ntp_servers=127.1.237.212:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 13:43:25.564512 6623 ts_manager.cc:194] Registered new tserver with Master: 47911ec8798141229a2269f150760a18 (127.1.237.194:44339)
W20260927 13:43:25.869997 6942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:25.870276 6942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:25.870426 6942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:25.877413 6942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:25.877526 6942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:25.877688 6942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:25.886152 6942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:25.887336 6942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:25.888911 6942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:25.901152 6948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.901192 6947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:25.901214 6950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:25.902333 6942 server_base.cc:1034] running on GCE node
I20260927 13:43:25.902940 6942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:25.903959 6942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:25.905179 6942 hybrid_clock.cc:648] HybridClock initialized: now 1790516605905134 us; error 44 us; skew 500 ppm
I20260927 13:43:25.905512 6942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:25.907763 6942 webserver.cc:460] Webserver started at http://127.1.237.195:45587/ using document root <none> and password file <none>
I20260927 13:43:25.908406 6942 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:25.908515 6942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:25.908828 6942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:25.911487 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "fc69573788834f2ab176e91c1a21e5f0"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "ec018b618ce5f66c5c9edea36a3b4f96"
server_key_iv: "39a9f3a5ab2fcf756101fbb96e2afbe4"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.912223 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "fc69573788834f2ab176e91c1a21e5f0"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "ec018b618ce5f66c5c9edea36a3b4f96"
server_key_iv: "39a9f3a5ab2fcf756101fbb96e2afbe4"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.917109 6942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 13:43:25.920193 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.921397 6942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:25.921569 6942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "fc69573788834f2ab176e91c1a21e5f0"
format_stamp: "Formatted at 2026-09-27 13:43:25 on dist-test-slave-7g3l"
server_key: "ec018b618ce5f66c5c9edea36a3b4f96"
server_key_iv: "39a9f3a5ab2fcf756101fbb96e2afbe4"
server_key_version: "encryptionkey@0"
I20260927 13:43:25.921797 6942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:25.927146 6942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:25.927886 6942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:25.928145 6942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:25.928941 6942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:25.930181 6942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:25.930295 6942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.930452 6942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:25.930538 6942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:25.961886 6942 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:42557
I20260927 13:43:25.961968 7068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:42557 every 8 connection(s)
I20260927 13:43:25.963126 6942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:25.970427 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.965528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:43059 (local address 127.1.237.254:36015)
0927 13:43:25.965709 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:25.965713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:25.966967 (+ 1254us) server_negotiation.cc:407] Connection header received
0927 13:43:25.968529 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:25.968533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:25.968578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:25.968670 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:25.969780 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:25.969784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:25.969787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:25.969801 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:25.969826 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:25.969837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:25.969929 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:25.970151 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 13:43:25.970193 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:43:25.970813 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 6942
I20260927 13:43:25.970961 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:25.971433 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.966026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36015 (local address 127.1.237.195:43059)
0927 13:43:25.966828 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:25.966861 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:25.968307 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:25.968851 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:25.968859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:25.968938 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:25.969530 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:25.969532 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:25.969565 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:25.969638 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:25.969917 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:25.969923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:25.969929 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:43:25.970219 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 13:43:25.970420 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":240,"threads_started":1}
I20260927 13:43:25.971455 1975 external_mini_cluster.cc:1428] Setting key c62ba14ba6cfdc4676b4f489401165bc
I20260927 13:43:25.972918 7069 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36015
I20260927 13:43:25.973227 7069 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:25.973786 7069 heartbeater.cc:511] Master 127.1.237.254:36015 requested a full tablet report, sending...
I20260927 13:43:25.975054 6622 ts_manager.cc:194] Registered new tserver with Master: fc69573788834f2ab176e91c1a21e5f0 (127.1.237.195:42557)
I20260927 13:43:25.986441 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:43:25.993211 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:25.990660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.1.237.254:36015)
0927 13:43:25.990824 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:25.990832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:25.991331 (+ 499us) server_negotiation.cc:407] Connection header received
0927 13:43:25.991519 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:25.991526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:25.991577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:25.991676 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:25.992594 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:25.992600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:25.992604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:25.992621 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:25.992658 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:25.992670 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:25.992806 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:25.992888 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 13:43:25.992942 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:25.997135 6622 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:26.003654 6622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:26.020560 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.015021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48664)
0927 13:43:26.017119 (+ 2098us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.017160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:26.017345 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:26.018286 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:26.018295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:26.018382 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:26.019200 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:26.019203 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:26.019228 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:26.019301 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:26.019793 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:26.019803 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:26.019810 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 13:43:26.019970 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 13:43:26.020076 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1880,"thread_start_us":2692,"threads_started":1}
I20260927 13:43:26.021610 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.015379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.1.237.193:42661)
0927 13:43:26.015963 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.015972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.017252 (+ 1280us) server_negotiation.cc:407] Connection header received
0927 13:43:26.017496 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.017505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.017748 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.017888 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.019474 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.019482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.019495 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.019536 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.019604 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.019625 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.019742 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.021161 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.021255 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":746,"threads_started":1}
I20260927 13:43:26.021626 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.016474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:44339 (local address 127.0.0.1:43314)
0927 13:43:26.017991 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.018006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:26.018098 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:26.019130 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:26.019133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:26.019193 (+ 60us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:26.020576 (+ 1383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:26.020579 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:26.020593 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:26.020646 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:26.020957 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:26.020962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:26.020970 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:43:26.021125 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 13:43:26.021169 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"thread_start_us":344,"threads_started":1}
I20260927 13:43:26.021839 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.015624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.1.237.194:44339)
0927 13:43:26.016168 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.016174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.018122 (+ 1948us) server_negotiation.cc:407] Connection header received
0927 13:43:26.018217 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.018228 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.018474 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.018638 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.020768 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.020774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.020785 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.020809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.020857 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.020872 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.021015 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.021083 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.021196 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":266,"threads_started":1}
I20260927 13:43:26.023000 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.015888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.1.237.195:42557)
0927 13:43:26.016526 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.016535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.019226 (+ 2691us) server_negotiation.cc:407] Connection header received
0927 13:43:26.019498 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.019507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.020038 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.020166 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.022118 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.022125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.022136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.022154 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.022211 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.022229 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.022370 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.022606 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.022685 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":775,"threads_started":1}
I20260927 13:43:26.025315 6874 tablet_service.cc:1467] Processing CreateTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 (DEFAULT_TABLE table=test-table [id=42020ca2aa1d41d6a24eb33ee6acd36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:26.025311 6744 tablet_service.cc:1467] Processing CreateTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 (DEFAULT_TABLE table=test-table [id=42020ca2aa1d41d6a24eb33ee6acd36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:26.026651 6874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585f1c6ef8284e9c9258f5e26ad38381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:26.026651 6744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585f1c6ef8284e9c9258f5e26ad38381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:26.027731 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.016575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:42557 (local address 127.0.0.1:54488)
0927 13:43:26.019109 (+ 2534us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.019126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:26.019303 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:26.021169 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:26.021174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:26.021194 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:26.021897 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:26.021902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:26.021916 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:26.021954 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:26.022459 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:26.022463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:26.022467 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 13:43:26.022575 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 13:43:26.027243 (+ 4668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"thread_start_us":2271,"threads_started":1}
I20260927 13:43:26.030483 7004 tablet_service.cc:1467] Processing CreateTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 (DEFAULT_TABLE table=test-table [id=42020ca2aa1d41d6a24eb33ee6acd36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:26.031633 7004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585f1c6ef8284e9c9258f5e26ad38381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:26.040907 7087 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Bootstrap starting.
I20260927 13:43:26.041756 7088 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Bootstrap starting.
I20260927 13:43:26.042819 7089 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Bootstrap starting.
I20260927 13:43:26.045667 7087 tablet_bootstrap.cc:654] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:26.045818 7089 tablet_bootstrap.cc:654] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:26.046399 7088 tablet_bootstrap.cc:654] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:26.046804 7089 log.cc:824] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:26.046871 7087 log.cc:824] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:26.047605 7088 log.cc:824] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:26.049819 7089 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: No bootstrap required, opened a new log
I20260927 13:43:26.050109 7088 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: No bootstrap required, opened a new log
I20260927 13:43:26.050098 7089 ts_tablet_manager.cc:1397] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 13:43:26.050343 7088 ts_tablet_manager.cc:1397] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260927 13:43:26.050611 7087 tablet_bootstrap.cc:492] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: No bootstrap required, opened a new log
I20260927 13:43:26.050848 7087 ts_tablet_manager.cc:1397] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 13:43:26.057418 7088 raft_consensus.cc:348] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.057814 7088 raft_consensus.cc:374] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:26.057916 7088 raft_consensus.cc:729] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47911ec8798141229a2269f150760a18, State: Initialized, Role: FOLLOWER
I20260927 13:43:26.058581 7088 consensus_queue.cc:260] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.059674 6939 heartbeater.cc:503] Master 127.1.237.254:36015 was elected leader, sending a full tablet report...
I20260927 13:43:26.060125 7088 ts_tablet_manager.cc:1428] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260927 13:43:26.060873 7087 raft_consensus.cc:348] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.061050 7089 raft_consensus.cc:348] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.061286 7087 raft_consensus.cc:374] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:26.061407 7087 raft_consensus.cc:729] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c873aaf559443959c6ccf0ff025d296, State: Initialized, Role: FOLLOWER
I20260927 13:43:26.061440 7089 raft_consensus.cc:374] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:26.061529 7089 raft_consensus.cc:729] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc69573788834f2ab176e91c1a21e5f0, State: Initialized, Role: FOLLOWER
I20260927 13:43:26.061957 7087 consensus_queue.cc:260] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.062156 7089 consensus_queue.cc:260] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.062853 6809 heartbeater.cc:503] Master 127.1.237.254:36015 was elected leader, sending a full tablet report...
I20260927 13:43:26.063431 7087 ts_tablet_manager.cc:1428] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260927 13:43:26.067829 7069 heartbeater.cc:503] Master 127.1.237.254:36015 was elected leader, sending a full tablet report...
I20260927 13:43:26.068354 7089 ts_tablet_manager.cc:1428] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
W20260927 13:43:26.135739 6810 tablet.cc:2369] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:43:26.214754 7070 tablet.cc:2369] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:43:26.302918 6940 tablet.cc:2369] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:26.386674 7093 raft_consensus.cc:482] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:26.386994 7093 raft_consensus.cc:504] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.388712 7093 leader_election.cc:288] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c873aaf559443959c6ccf0ff025d296 (127.1.237.193:42661), fc69573788834f2ab176e91c1a21e5f0 (127.1.237.195:42557)
I20260927 13:43:26.391716 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.388967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:50051 (local address 127.1.237.193:42661)
0927 13:43:26.389139 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.389147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.389700 (+ 553us) server_negotiation.cc:407] Connection header received
0927 13:43:26.389921 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.389926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.389975 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.390080 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.391058 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.391064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.391067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.391086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.391123 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.391134 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.391281 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.391368 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.391428 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 13:43:26.391813 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.389290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52821 (local address 127.1.237.195:42557)
0927 13:43:26.389484 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.389492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.390074 (+ 582us) server_negotiation.cc:407] Connection header received
0927 13:43:26.390236 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.390242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.390295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.390396 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.391188 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.391193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.391198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.391215 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.391253 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.391264 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.391385 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.391492 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.391551 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:43:26.392043 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.389144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:42557 (local address 127.1.237.194:52821)
0927 13:43:26.389972 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.389985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:26.390111 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:26.390419 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:26.390422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:26.390443 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:26.390978 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:26.390981 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:26.390992 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:26.391026 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:26.391365 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:26.391371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:26.391376 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:43:26.391546 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 13:43:26.391592 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":518,"threads_started":1}
I20260927 13:43:26.392233 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.389002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.1.237.194:50051)
0927 13:43:26.389593 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.389607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:26.389761 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:26.390103 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:26.390108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:26.390131 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:26.390812 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:43:26.390817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:26.390837 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:43:26.390888 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:26.391257 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:26.391261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:26.391266 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:43:26.391364 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:43:26.391417 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":327,"threads_started":1}
I20260927 13:43:26.393033 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585f1c6ef8284e9c9258f5e26ad38381" candidate_uuid: "47911ec8798141229a2269f150760a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc69573788834f2ab176e91c1a21e5f0" is_pre_election: true
I20260927 13:43:26.393092 6764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585f1c6ef8284e9c9258f5e26ad38381" candidate_uuid: "47911ec8798141229a2269f150760a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c873aaf559443959c6ccf0ff025d296" is_pre_election: true
I20260927 13:43:26.393481 7024 raft_consensus.cc:2454] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47911ec8798141229a2269f150760a18 in term 0.
I20260927 13:43:26.393563 6764 raft_consensus.cc:2454] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47911ec8798141229a2269f150760a18 in term 0.
I20260927 13:43:26.394104 6830 leader_election.cc:302] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47911ec8798141229a2269f150760a18, fc69573788834f2ab176e91c1a21e5f0; no voters:
I20260927 13:43:26.394470 7093 raft_consensus.cc:2781] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:26.394595 7093 raft_consensus.cc:482] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:26.394686 7093 raft_consensus.cc:3037] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:26.396939 7093 raft_consensus.cc:504] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.397564 7093 leader_election.cc:288] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [CANDIDATE]: Term 1 election: Requested vote from peers 9c873aaf559443959c6ccf0ff025d296 (127.1.237.193:42661), fc69573788834f2ab176e91c1a21e5f0 (127.1.237.195:42557)
I20260927 13:43:26.397946 6764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585f1c6ef8284e9c9258f5e26ad38381" candidate_uuid: "47911ec8798141229a2269f150760a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c873aaf559443959c6ccf0ff025d296"
I20260927 13:43:26.398138 6764 raft_consensus.cc:3037] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:26.398113 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585f1c6ef8284e9c9258f5e26ad38381" candidate_uuid: "47911ec8798141229a2269f150760a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc69573788834f2ab176e91c1a21e5f0"
I20260927 13:43:26.398305 7024 raft_consensus.cc:3037] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:26.400285 6764 raft_consensus.cc:2454] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47911ec8798141229a2269f150760a18 in term 1.
I20260927 13:43:26.400465 7024 raft_consensus.cc:2454] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47911ec8798141229a2269f150760a18 in term 1.
I20260927 13:43:26.400689 6827 leader_election.cc:302] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47911ec8798141229a2269f150760a18, 9c873aaf559443959c6ccf0ff025d296; no voters:
I20260927 13:43:26.400977 7093 raft_consensus.cc:2781] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:26.401365 7093 raft_consensus.cc:686] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 1 LEADER]: Becoming Leader. State: Replica: 47911ec8798141229a2269f150760a18, State: Running, Role: LEADER
I20260927 13:43:26.401785 7093 consensus_queue.cc:237] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } }
I20260927 13:43:26.405818 6621 catalog_manager.cc:5495] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47911ec8798141229a2269f150760a18 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "47911ec8798141229a2269f150760a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47911ec8798141229a2269f150760a18" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:26.476248 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.470701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.1.237.194:44339)
0927 13:43:26.470849 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.470853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.470976 (+ 123us) server_negotiation.cc:407] Connection header received
0927 13:43:26.471237 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.471242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.471333 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.471416 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.475689 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.475693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.475696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.475712 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.475743 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.475754 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.475855 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.476002 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.476067 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:43:26.483842 7024 raft_consensus.cc:1261] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 1 FOLLOWER]: Refusing update from remote peer 47911ec8798141229a2269f150760a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:26.483985 6764 raft_consensus.cc:1261] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 1 FOLLOWER]: Refusing update from remote peer 47911ec8798141229a2269f150760a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:26.484836 7098 consensus_queue.cc:1035] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc69573788834f2ab176e91c1a21e5f0" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 42557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:26.485144 7093 consensus_queue.cc:1035] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c873aaf559443959c6ccf0ff025d296" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:26.496188 7101 mvcc.cc:205] Tried to move back new op lower bound from 7333956020148797440 to 7333956019825143808. Current Snapshot: MvccSnapshot[applied={T|T < 7333956020148797440}]
I20260927 13:43:26.498173 7103 mvcc.cc:205] Tried to move back new op lower bound from 7333956020148797440 to 7333956019825143808. Current Snapshot: MvccSnapshot[applied={T|T < 7333956020148797440}]
I20260927 13:43:26.501500 7102 mvcc.cc:205] Tried to move back new op lower bound from 7333956020148797440 to 7333956019825143808. Current Snapshot: MvccSnapshot[applied={T|T < 7333956020148797440}]
I20260927 13:43:26.510449 6623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:26.510684 6623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:26.514245 6623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1706f2f4524b4d6eb40a1ab07a644aa8: Sending DeleteTablet for 3 replicas of tablet 585f1c6ef8284e9c9258f5e26ad38381
I20260927 13:43:26.515368 6744 tablet_service.cc:1514] Processing DeleteTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:26 UTC) from {username='slave'} at 127.0.0.1:48664
I20260927 13:43:26.515419 7004 tablet_service.cc:1514] Processing DeleteTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:26 UTC) from {username='slave'} at 127.0.0.1:54488
I20260927 13:43:26.515623 6874 tablet_service.cc:1514] Processing DeleteTablet for tablet 585f1c6ef8284e9c9258f5e26ad38381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:26 UTC) from {username='slave'} at 127.0.0.1:43314
I20260927 13:43:26.517771 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6682
I20260927 13:43:26.545128 7112 tablet_replica.cc:330] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: stopping tablet replica
I20260927 13:43:26.547096 7112 raft_consensus.cc:2229] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:26.548458 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.519258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.1.237.193:42661)
0927 13:43:26.519460 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.519468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.543460 (+ 23992us) server_negotiation.cc:407] Connection header received
0927 13:43:26.544069 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.544079 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.544233 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.544327 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.547472 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.547482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.547507 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.547550 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.547602 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.547624 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.547763 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.548039 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.548117 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:43:26.549731 6804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:26.550252 7112 raft_consensus.cc:2258] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:26.551134 7111 tablet_replica.cc:330] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: stopping tablet replica
I20260927 13:43:26.551594 7111 raft_consensus.cc:2229] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:26.552109 7111 raft_consensus.cc:2258] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:26.553097 7112 ts_tablet_manager.cc:1905] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:26.554021 7111 ts_tablet_manager.cc:1905] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:26.560762 7112 ts_tablet_manager.cc:1918] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:26.560853 7111 ts_tablet_manager.cc:1918] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:26.560985 7111 log.cc:1196] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/585f1c6ef8284e9c9258f5e26ad38381
I20260927 13:43:26.561180 7112 log.cc:1196] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/585f1c6ef8284e9c9258f5e26ad38381
I20260927 13:43:26.561622 7111 ts_tablet_manager.cc:1939] T 585f1c6ef8284e9c9258f5e26ad38381 P fc69573788834f2ab176e91c1a21e5f0: Deleting consensus metadata
I20260927 13:43:26.561995 7112 ts_tablet_manager.cc:1939] T 585f1c6ef8284e9c9258f5e26ad38381 P 47911ec8798141229a2269f150760a18: Deleting consensus metadata
I20260927 13:43:26.563691 6608 catalog_manager.cc:4841] TS 47911ec8798141229a2269f150760a18 (127.1.237.194:44339): tablet 585f1c6ef8284e9c9258f5e26ad38381 (table test-table [id=42020ca2aa1d41d6a24eb33ee6acd36a]) successfully deleted
I20260927 13:43:26.568313 6610 catalog_manager.cc:4841] TS fc69573788834f2ab176e91c1a21e5f0 (127.1.237.195:42557): tablet 585f1c6ef8284e9c9258f5e26ad38381 (table test-table [id=42020ca2aa1d41d6a24eb33ee6acd36a]) successfully deleted
I20260927 13:43:26.656694 7110 tablet_replica.cc:330] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296: stopping tablet replica
I20260927 13:43:26.657135 7110 raft_consensus.cc:2229] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:26.657604 7110 raft_consensus.cc:2258] T 585f1c6ef8284e9c9258f5e26ad38381 P 9c873aaf559443959c6ccf0ff025d296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:26.658453 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6682
I20260927 13:43:26.676437 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6812
W20260927 13:43:26.676514 6607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:42661 (error 108)
I20260927 13:43:26.678632 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.677786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48678)
0927 13:43:26.678303 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.678410 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":339,"threads_started":1}
W20260927 13:43:26.678876 6607 catalog_manager.cc:4584] TS 9c873aaf559443959c6ccf0ff025d296 (127.1.237.193:42661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 585f1c6ef8284e9c9258f5e26ad38381: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
I20260927 13:43:26.679720 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.677431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.1.237.194:44339)
0927 13:43:26.677595 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.677601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.677620 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:43:26.677704 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.677709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.677766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.677866 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.678943 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.678954 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.678959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.678974 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.679004 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.679015 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.679188 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.679412 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.679480 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:43:26.680305 6934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:26.732577 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.732106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48680)
0927 13:43:26.732308 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.732406 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:43:26.733569 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.733191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48684)
0927 13:43:26.733332 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.733409 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 13:43:26.786414 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6812
I20260927 13:43:26.793908 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.793490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48692)
0927 13:43:26.793705 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.793785 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 13:43:26.795019 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.794628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48702)
0927 13:43:26.794776 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.794859 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:43:26.803995 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6942
I20260927 13:43:26.806447 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.804694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.1.237.195:42557)
0927 13:43:26.804853 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:26.804858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:26.804875 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:26.804954 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:26.804958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:26.805065 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:26.805162 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:26.805872 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:26.805876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:26.805879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:26.805892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:26.805920 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:26.805931 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:26.806048 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:26.806081 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 13:43:26.806120 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:43:26.807080 7064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:26.875699 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.875218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48716)
0927 13:43:26.875428 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.875542 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:43:26.876755 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:26.876304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42661 (local address 127.0.0.1:48720)
0927 13:43:26.876492 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:26.876579 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 13:43:26.889513 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6942
I20260927 13:43:26.904960 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 6592
I20260927 13:43:26.905994 6653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:27.008671 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 6592
2026-09-27T13:43:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2807 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 13:43:27.048691 1975 mini_oidc.cc:152] Starting JWKS server
I20260927 13:43:27.051067 1975 webserver.cc:460] Webserver started at https://127.0.0.1:42133/ using document root <none> and password file <none>
I20260927 13:43:27.051378 1975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:43:27.051985 1975 webserver.cc:460] Webserver started at http://127.0.0.1:44723/ using document root <none> and password file <none>
2026-09-27T13:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:27Z Disabled control of system clock
I20260927 13:43:27.075912 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:40123
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:40123
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42133/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/testchainca.cert with env {}
W20260927 13:43:27.372206 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:27.372439 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:27.372515 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:27.378931 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:27.379031 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:27.379066 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:27.379118 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:27.379168 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:43:27.379225 7125 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:43:27.379263 7125 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42133/jwks.json
I20260927 13:43:27.387904 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:40123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42133/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:27.388947 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:27.389971 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:27.400870 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:27.400827 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:27.400923 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:27.401711 7125 server_base.cc:1034] running on GCE node
I20260927 13:43:27.402377 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:27.403662 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:27.404870 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790516607404853 us; error 43 us; skew 500 ppm
I20260927 13:43:27.405195 7125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:27.407313 7125 webserver.cc:460] Webserver started at http://127.1.237.254:43087/ using document root <none> and password file <none>
I20260927 13:43:27.407927 7125 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:27.408041 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:27.408305 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:27.410903 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "5216c0ac577e44df920b15cfbfe4db90"
format_stamp: "Formatted at 2026-09-27 13:43:27 on dist-test-slave-7g3l"
server_key: "7f4e3a2813aa79bccd9c371e088c4989"
server_key_iv: "f6a054f968fe24b4e6a77b382368b3a6"
server_key_version: "encryptionkey@0"
I20260927 13:43:27.411684 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "5216c0ac577e44df920b15cfbfe4db90"
format_stamp: "Formatted at 2026-09-27 13:43:27 on dist-test-slave-7g3l"
server_key: "7f4e3a2813aa79bccd9c371e088c4989"
server_key_iv: "f6a054f968fe24b4e6a77b382368b3a6"
server_key_version: "encryptionkey@0"
I20260927 13:43:27.416057 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:43:27.419142 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:27.420305 7125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:43:27.420475 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "5216c0ac577e44df920b15cfbfe4db90"
format_stamp: "Formatted at 2026-09-27 13:43:27 on dist-test-slave-7g3l"
server_key: "7f4e3a2813aa79bccd9c371e088c4989"
server_key_iv: "f6a054f968fe24b4e6a77b382368b3a6"
server_key_version: "encryptionkey@0"
I20260927 13:43:27.420667 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:27.437062 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:27.437952 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:27.438264 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:27.458083 7125 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:40123
I20260927 13:43:27.458104 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:40123 every 8 connection(s)
I20260927 13:43:27.459334 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:27.460067 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7125
I20260927 13:43:27.460290 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:27.461035 1975 external_mini_cluster.cc:1428] Setting key 5564100239805396e7b61d3422a663a3
I20260927 13:43:27.464439 7191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:27.472983 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:27.462625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.1.237.254:40123)
0927 13:43:27.463421 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:27.463432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:27.463467 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:43:27.465094 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:27.465113 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:27.465495 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:27.465830 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:27.466363 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.467710 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.468971 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.469561 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.469740 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:27.470853 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:27.470861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:27.470872 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:27.471448 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:27.471517 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:27.471543 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:27.471672 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:27.471895 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 13:43:27.472177 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":226,"threads_started":1}
I20260927 13:43:27.474050 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90: Bootstrap starting.
I20260927 13:43:27.476321 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:43:27.477015 7191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:27.477883 7191 log.cc:824] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:27.480489 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90: No bootstrap required, opened a new log
I20260927 13:43:27.487383 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } }
I20260927 13:43:27.487687 7191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:27.487793 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5216c0ac577e44df920b15cfbfe4db90, State: Initialized, Role: FOLLOWER
I20260927 13:43:27.488283 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } }
I20260927 13:43:27.488498 7191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:27.488637 7191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:27.488817 7191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:27.491118 7191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } }
I20260927 13:43:27.491632 7191 leader_election.cc:302] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5216c0ac577e44df920b15cfbfe4db90; no voters:
I20260927 13:43:27.492162 7191 leader_election.cc:288] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:27.492311 7196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:27.492990 7196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [term 1 LEADER]: Becoming Leader. State: Replica: 5216c0ac577e44df920b15cfbfe4db90, State: Running, Role: LEADER
I20260927 13:43:27.493455 7196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } }
I20260927 13:43:27.494124 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:27.496261 7197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5216c0ac577e44df920b15cfbfe4db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } } }
I20260927 13:43:27.496176 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5216c0ac577e44df920b15cfbfe4db90. Latest consensus state: current_term: 1 leader_uuid: "5216c0ac577e44df920b15cfbfe4db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5216c0ac577e44df920b15cfbfe4db90" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40123 } } }
I20260927 13:43:27.496523 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:27.496528 7197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:27.498489 7201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:27.504050 7201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:27.511354 7201 catalog_manager.cc:1347] Generated new cluster ID: 089f32a74a36455cafc9749600f2b8b4
I20260927 13:43:27.511461 7201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:27.537904 7201 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:27.539105 7201 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:27.548081 7201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5216c0ac577e44df920b15cfbfe4db90: Generated new TSK 0
I20260927 13:43:27.548787 7201 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:43:27.584226 7221 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 13:43:27.584363 7221 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:43:27.585417 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:27.565024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.1.237.254:40123)
0927 13:43:27.565186 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:27.565192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:27.565730 (+ 538us) server_negotiation.cc:407] Connection header received
0927 13:43:27.565976 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:27.565982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:27.566069 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:27.566204 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:43:27.566645 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.567753 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.569099 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.569722 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.569833 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:27.570328 (+ 495us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:43:27.584786 (+ 14458us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:43:27.585087 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 13:43:27.585528 7194 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43976: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 13:43:27.586637 7220 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:27.565033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:40123 (local address 127.0.0.1:43976)
0927 13:43:27.565639 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:27.565656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:27.565778 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:27.566227 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:27.566230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:27.566238 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:43:27.566513 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:27.566520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.567859 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.567862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:27.568956 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:27.568977 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:27.569874 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:27.569878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:27.569985 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:27.570043 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:43:27.584990 (+ 14947us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 13:43:27.585233 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.237.254:40123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":309,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.237.254:40123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42133/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 13:43:27.588807 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7125
I20260927 13:43:27.589787 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:27.679324 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7125
2026-09-27T13:43:27Z chronyd exiting
I20260927 13:43:27.728080 1975 test_util.cc:175] -----------------------------------------------
I20260927 13:43:27.728235 1975 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790516567150638-1975-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (683 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 13:43:27.731690 1975 mini_oidc.cc:152] Starting JWKS server
I20260927 13:43:27.733381 1975 webserver.cc:460] Webserver started at https://127.0.0.1:37131/ using document root <none> and password file <none>
I20260927 13:43:27.733613 1975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:43:27.734259 1975 webserver.cc:460] Webserver started at http://127.0.0.1:41333/ using document root <none> and password file <none>
2026-09-27T13:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:27Z Disabled control of system clock
I20260927 13:43:27.759582 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36929
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/testchainca.cert with env {}
W20260927 13:43:28.055347 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:28.055605 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:28.055769 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:28.062548 7228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:28.062655 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:28.062757 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:28.062852 7228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:28.062934 7228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:43:28.063040 7228 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:43:28.063100 7228 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37131/jwks.json
I20260927 13:43:28.072008 7228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:28.073171 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:28.074360 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:28.085131 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:28.085147 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:28.085147 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:28.085916 7228 server_base.cc:1034] running on GCE node
I20260927 13:43:28.086653 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:28.087958 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:28.089166 7228 hybrid_clock.cc:648] HybridClock initialized: now 1790516608089138 us; error 43 us; skew 500 ppm
I20260927 13:43:28.089498 7228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:28.091663 7228 webserver.cc:460] Webserver started at http://127.1.237.254:44405/ using document root <none> and password file <none>
I20260927 13:43:28.092289 7228 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:28.092415 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:28.092710 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:28.095263 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "315a7c6216c444ada0e6ef164d5f5caa"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "f533d9d67b3cf46c4758eea7b46082bc"
server_key_iv: "8517285003874dd2cda53c864952a0c8"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.095984 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "315a7c6216c444ada0e6ef164d5f5caa"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "f533d9d67b3cf46c4758eea7b46082bc"
server_key_iv: "8517285003874dd2cda53c864952a0c8"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.100445 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 13:43:28.103577 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:28.104818 7228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 13:43:28.104995 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "315a7c6216c444ada0e6ef164d5f5caa"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "f533d9d67b3cf46c4758eea7b46082bc"
server_key_iv: "8517285003874dd2cda53c864952a0c8"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.105204 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:28.122203 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:28.122889 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:28.123209 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:28.140439 7228 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36929
I20260927 13:43:28.140450 7293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36929 every 8 connection(s)
I20260927 13:43:28.141623 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:28.143323 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7228
I20260927 13:43:28.143513 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:28.144270 1975 external_mini_cluster.cc:1428] Setting key df19f3fc5116de466d72c48d9e4aa896
I20260927 13:43:28.145799 7294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:28.158159 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:28.146223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.1.237.254:36929)
0927 13:43:28.147319 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:28.147332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:28.147521 (+ 189us) server_negotiation.cc:407] Connection header received
0927 13:43:28.149030 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:28.149054 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:28.149611 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:28.150021 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:28.150489 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.152403 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.153539 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.154467 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.154635 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.155669 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:28.155679 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:28.155695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:28.156363 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:28.156448 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:28.156481 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:28.156613 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:28.156737 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 13:43:28.157042 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":364,"threads_started":1}
I20260927 13:43:28.159456 7294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa: Bootstrap starting.
I20260927 13:43:28.161579 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:43:28.162591 7294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:28.163718 7294 log.cc:824] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:28.166285 7294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa: No bootstrap required, opened a new log
I20260927 13:43:28.172712 7294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } }
I20260927 13:43:28.172981 7294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:28.173058 7294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315a7c6216c444ada0e6ef164d5f5caa, State: Initialized, Role: FOLLOWER
I20260927 13:43:28.173466 7294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } }
I20260927 13:43:28.173630 7294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:28.173714 7294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:28.173827 7294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:28.175668 7294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } }
I20260927 13:43:28.176110 7294 leader_election.cc:302] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 315a7c6216c444ada0e6ef164d5f5caa; no voters:
I20260927 13:43:28.176563 7294 leader_election.cc:288] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:28.176740 7299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:28.177482 7299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [term 1 LEADER]: Becoming Leader. State: Replica: 315a7c6216c444ada0e6ef164d5f5caa, State: Running, Role: LEADER
I20260927 13:43:28.178020 7299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } }
I20260927 13:43:28.178493 7294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:28.181010 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "315a7c6216c444ada0e6ef164d5f5caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } } }
I20260927 13:43:28.181169 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 315a7c6216c444ada0e6ef164d5f5caa. Latest consensus state: current_term: 1 leader_uuid: "315a7c6216c444ada0e6ef164d5f5caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "315a7c6216c444ada0e6ef164d5f5caa" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36929 } } }
I20260927 13:43:28.181847 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:28.181893 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:28.188212 7307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:28.192147 7307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:28.199854 7307 catalog_manager.cc:1347] Generated new cluster ID: 3230d494876d4b25b6f62dd94c203ab4
I20260927 13:43:28.199972 7307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:28.222813 7307 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:28.224313 7307 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:28.235957 7307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 315a7c6216c444ada0e6ef164d5f5caa: Generated new TSK 0
I20260927 13:43:28.236706 7307 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 13:43:28.270710 7324 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 13:43:28.270852 7324 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:43:28.271787 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:28.253425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.1.237.254:36929)
0927 13:43:28.253588 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:28.253595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:28.254114 (+ 519us) server_negotiation.cc:407] Connection header received
0927 13:43:28.254303 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:28.254306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:28.254366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:28.254474 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:43:28.254984 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.255729 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.257637 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.258030 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.258119 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.258583 (+ 464us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:43:28.271328 (+ 12745us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:43:28.271564 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 13:43:28.271911 7297 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 13:43:28.272122 7323 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:28.253287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36929 (local address 127.0.0.1:47392)
0927 13:43:28.254008 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:28.254027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:28.254160 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:28.254509 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:28.254514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:28.254527 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:43:28.254808 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:28.254819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.255850 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.255856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:28.257516 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:28.257530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.258168 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.258174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:28.258351 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.258391 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:43:28.271509 (+ 13118us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 13:43:28.271737 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.237.254:36929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":421,"threads_started":1}
I20260927 13:43:28.272984 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7228
I20260927 13:43:28.273905 7289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:28.362341 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7228
2026-09-27T13:43:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 13:43:28.416218 1975 mini_oidc.cc:152] Starting JWKS server
I20260927 13:43:28.417833 1975 webserver.cc:460] Webserver started at https://127.0.0.1:37697/ using document root <none> and password file <none>
I20260927 13:43:28.418064 1975 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 13:43:28.418602 1975 webserver.cc:460] Webserver started at http://127.0.0.1:37135/ using document root <none> and password file <none>
2026-09-27T13:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:28Z Disabled control of system clock
I20260927 13:43:28.440896 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36563
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:43:28.736109 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:28.736359 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:28.736445 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:28.743018 7331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:28.743096 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:28.743130 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:28.743240 7331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:28.743285 7331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:43:28.743353 7331 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:43:28.743391 7331 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37697/jwks.json
I20260927 13:43:28.751634 7331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:28.752688 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:28.754200 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:28.765136 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:28.765204 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:28.765223 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:28.765928 7331 server_base.cc:1034] running on GCE node
I20260927 13:43:28.766593 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:28.767846 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:28.769033 7331 hybrid_clock.cc:648] HybridClock initialized: now 1790516608768996 us; error 28 us; skew 500 ppm
I20260927 13:43:28.769354 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:28.771348 7331 webserver.cc:460] Webserver started at http://127.1.237.254:42665/ using document root <none> and password file <none>
I20260927 13:43:28.771966 7331 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:28.772078 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:28.772401 7331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:28.774940 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "e804d28c507b47c790d48decfce9145d"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "0e8ddc149cb3b2df78f39bfa18431c03"
server_key_iv: "1996b991889c2c3f9a5fb775c07b51d4"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.775655 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "e804d28c507b47c790d48decfce9145d"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "0e8ddc149cb3b2df78f39bfa18431c03"
server_key_iv: "1996b991889c2c3f9a5fb775c07b51d4"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.780114 7331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:28.783213 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:28.784456 7331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:43:28.784624 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "e804d28c507b47c790d48decfce9145d"
format_stamp: "Formatted at 2026-09-27 13:43:28 on dist-test-slave-7g3l"
server_key: "0e8ddc149cb3b2df78f39bfa18431c03"
server_key_iv: "1996b991889c2c3f9a5fb775c07b51d4"
server_key_version: "encryptionkey@0"
I20260927 13:43:28.784839 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:28.811605 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:28.812438 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:28.812736 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:28.830005 7331 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36563
I20260927 13:43:28.830029 7396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36563 every 8 connection(s)
I20260927 13:43:28.831284 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:28.834008 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7331
I20260927 13:43:28.834201 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:28.834944 1975 external_mini_cluster.cc:1428] Setting key 24a7f63eb69998f552d9b1d032693629
I20260927 13:43:28.835120 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:28.847301 7397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d: Bootstrap starting.
I20260927 13:43:28.847862 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:28.837403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.1.237.254:36563)
0927 13:43:28.838373 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:28.838383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:28.838423 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:43:28.839977 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:28.839998 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:28.840417 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:28.840789 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:28.841306 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.842538 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.843828 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.844459 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.844686 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.845761 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:28.845767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:28.845778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:28.846240 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:28.846304 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:28.846340 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:28.846457 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:28.846658 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 13:43:28.846925 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":338,"threads_started":1}
I20260927 13:43:28.850345 7397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:28.851305 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:43:28.851385 7397 log.cc:824] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:28.853996 7397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d: No bootstrap required, opened a new log
I20260927 13:43:28.860757 7397 raft_consensus.cc:348] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } }
I20260927 13:43:28.861073 7397 raft_consensus.cc:374] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:28.861182 7397 raft_consensus.cc:729] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e804d28c507b47c790d48decfce9145d, State: Initialized, Role: FOLLOWER
I20260927 13:43:28.861755 7397 consensus_queue.cc:260] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } }
I20260927 13:43:28.861972 7397 raft_consensus.cc:388] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:28.862121 7397 raft_consensus.cc:482] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:28.862299 7397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:28.864477 7397 raft_consensus.cc:504] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } }
I20260927 13:43:28.864919 7397 leader_election.cc:302] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e804d28c507b47c790d48decfce9145d; no voters:
I20260927 13:43:28.865419 7397 leader_election.cc:288] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:28.865551 7402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:28.866168 7402 raft_consensus.cc:686] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [term 1 LEADER]: Becoming Leader. State: Replica: e804d28c507b47c790d48decfce9145d, State: Running, Role: LEADER
I20260927 13:43:28.866662 7402 consensus_queue.cc:237] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } }
I20260927 13:43:28.867312 7397 sys_catalog.cc:564] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:28.870250 7404 sys_catalog.cc:455] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e804d28c507b47c790d48decfce9145d. Latest consensus state: current_term: 1 leader_uuid: "e804d28c507b47c790d48decfce9145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } } }
I20260927 13:43:28.870658 7404 sys_catalog.cc:458] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:28.870784 7403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e804d28c507b47c790d48decfce9145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e804d28c507b47c790d48decfce9145d" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36563 } } }
I20260927 13:43:28.871210 7403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:28.871614 7413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:28.875361 7413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:28.882840 7413 catalog_manager.cc:1347] Generated new cluster ID: b2113a858b0b4dcea79b31d75eb5910f
I20260927 13:43:28.882993 7413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:28.896289 7413 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:28.897395 7413 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:28.902218 7413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e804d28c507b47c790d48decfce9145d: Generated new TSK 0
I20260927 13:43:28.902896 7413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:43:28.928207 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:28.920364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.1.237.254:36563)
0927 13:43:28.920526 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:28.920532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:28.920838 (+ 306us) server_negotiation.cc:407] Connection header received
0927 13:43:28.921052 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:28.921058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:28.921140 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:28.921293 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:43:28.921690 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.922788 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.924098 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.924764 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.924882 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.925351 (+ 469us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:43:28.927617 (+ 2266us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:43:28.927914 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45122: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 13:43:28.928330 7400 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45122: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 13:43:28.929912 7426 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:28.920196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36563 (local address 127.0.0.1:45122)
0927 13:43:28.920736 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:28.920751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:28.920864 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:28.921309 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:28.921313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:28.921322 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:43:28.921549 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:28.921557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.922917 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.922920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:28.923968 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:28.923979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:28.924965 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:28.924969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:28.925084 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:28.925111 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:43:28.927795 (+ 2684us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 13:43:28.929543 (+ 1748us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.237.254:36563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1477,"thread_start_us":297,"threads_started":1}
I20260927 13:43:28.931015 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7331
I20260927 13:43:28.932142 7392 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 13:43:28.940393 7427 webserver.cc:533] Webserver: sslize failed:
E20260927 13:43:28.940465 7427 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 13:43:29.020211 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7331
2026-09-27T13:43:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (657 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T13:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:29Z Disabled control of system clock
I20260927 13:43:29.097808 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38083
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:38083
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/testchainca.cert with env {}
W20260927 13:43:29.396030 7432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:29.396292 7432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:29.396374 7432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:29.403095 7432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:29.403196 7432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:29.403235 7432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:29.403288 7432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:29.403333 7432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 13:43:29.403386 7432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 13:43:29.403424 7432 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 13:43:29.412261 7432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:38083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:29.413250 7432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:29.414372 7432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:29.425175 7437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:29.425220 7438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:29.425122 7440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:29.426179 7432 server_base.cc:1034] running on GCE node
I20260927 13:43:29.426874 7432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:29.428113 7432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:29.429328 7432 hybrid_clock.cc:648] HybridClock initialized: now 1790516609429266 us; error 50 us; skew 500 ppm
I20260927 13:43:29.429649 7432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:43:29.431807 7432 webserver.cc:460] Webserver started at http://127.1.237.254:42923/ using document root <none> and password file <none>
I20260927 13:43:29.432451 7432 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:29.432564 7432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:29.432857 7432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:29.435685 7432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "db75d4598db84a46a6351dbd84efa3a9"
format_stamp: "Formatted at 2026-09-27 13:43:29 on dist-test-slave-7g3l"
server_key: "05338d5f46bc188923b2848eef522977"
server_key_iv: "1ecef19002dbab7ce7d854afdd5e15d9"
server_key_version: "encryptionkey@0"
I20260927 13:43:29.436851 7432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "db75d4598db84a46a6351dbd84efa3a9"
format_stamp: "Formatted at 2026-09-27 13:43:29 on dist-test-slave-7g3l"
server_key: "05338d5f46bc188923b2848eef522977"
server_key_iv: "1ecef19002dbab7ce7d854afdd5e15d9"
server_key_version: "encryptionkey@0"
I20260927 13:43:29.442934 7432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 13:43:29.445940 7446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:29.447136 7432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:29.447357 7432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "db75d4598db84a46a6351dbd84efa3a9"
format_stamp: "Formatted at 2026-09-27 13:43:29 on dist-test-slave-7g3l"
server_key: "05338d5f46bc188923b2848eef522977"
server_key_iv: "1ecef19002dbab7ce7d854afdd5e15d9"
server_key_version: "encryptionkey@0"
I20260927 13:43:29.447590 7432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:29.462906 7432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:29.463634 7432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:29.463904 7432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:29.481258 7432 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:38083
I20260927 13:43:29.481278 7497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:38083 every 8 connection(s)
I20260927 13:43:29.482540 7432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:29.485841 7498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:29.492241 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7432
I20260927 13:43:29.492457 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:29.493080 1975 external_mini_cluster.cc:1428] Setting key 2f19a7756c9632a30998aea4c578035d
I20260927 13:43:29.495280 7498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9: Bootstrap starting.
I20260927 13:43:29.498138 7498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:29.499253 7498 log.cc:824] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:29.502285 7498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9: No bootstrap required, opened a new log
I20260927 13:43:29.506209 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:29.494696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.1.237.254:38083)
0927 13:43:29.495733 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:29.495743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:29.495779 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:43:29.497343 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:29.497363 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:29.497753 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:29.498078 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:29.498655 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.500356 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.502165 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.502857 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.502991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:29.504134 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:29.504143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:29.504158 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:43:29.504647 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:29.504721 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:43:29.504739 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:29.504826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:29.505058 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 13:43:29.505361 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":467,"threads_started":1}
I20260927 13:43:29.509610 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:43:29.510259 7498 raft_consensus.cc:348] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } }
I20260927 13:43:29.510563 7498 raft_consensus.cc:374] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:29.510653 7498 raft_consensus.cc:729] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db75d4598db84a46a6351dbd84efa3a9, State: Initialized, Role: FOLLOWER
I20260927 13:43:29.511200 7498 consensus_queue.cc:260] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } }
I20260927 13:43:29.511416 7498 raft_consensus.cc:388] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:29.511561 7498 raft_consensus.cc:482] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:29.511729 7498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:29.513989 7498 raft_consensus.cc:504] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } }
I20260927 13:43:29.514415 7498 leader_election.cc:302] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db75d4598db84a46a6351dbd84efa3a9; no voters:
I20260927 13:43:29.514971 7498 leader_election.cc:288] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:29.515141 7503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:29.516686 7503 raft_consensus.cc:686] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [term 1 LEADER]: Becoming Leader. State: Replica: db75d4598db84a46a6351dbd84efa3a9, State: Running, Role: LEADER
I20260927 13:43:29.516963 7498 sys_catalog.cc:564] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:29.517198 7503 consensus_queue.cc:237] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } }
I20260927 13:43:29.520201 7505 sys_catalog.cc:455] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db75d4598db84a46a6351dbd84efa3a9. Latest consensus state: current_term: 1 leader_uuid: "db75d4598db84a46a6351dbd84efa3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } } }
I20260927 13:43:29.520623 7505 sys_catalog.cc:458] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:29.521319 7504 sys_catalog.cc:455] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db75d4598db84a46a6351dbd84efa3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db75d4598db84a46a6351dbd84efa3a9" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38083 } } }
I20260927 13:43:29.521857 7504 sys_catalog.cc:458] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:29.522720 7510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:29.527931 7510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:29.535636 7510 catalog_manager.cc:1347] Generated new cluster ID: 9f3a0068fc174df6a7656103dd986ba5
I20260927 13:43:29.535732 7510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:29.547194 7510 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:29.548298 7510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:29.556576 7510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db75d4598db84a46a6351dbd84efa3a9: Generated new TSK 0
I20260927 13:43:29.557224 7510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:43:29.610796 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:29.577113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.1.237.254:38083)
0927 13:43:29.577290 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:29.577296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:29.577592 (+ 296us) server_negotiation.cc:407] Connection header received
0927 13:43:29.577779 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:29.577784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:29.577880 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:29.578040 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0927 13:43:29.578445 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.579634 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.581023 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.581704 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.581841 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:29.582271 (+ 430us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 13:43:29.610090 (+ 27819us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:43:29.610454 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51920: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 13:43:29.610842 7527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:29.576958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38083 (local address 127.0.0.1:51920)
0927 13:43:29.577497 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:29.577513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:29.577622 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:29.578054 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:29.578057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:29.578066 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 13:43:29.578313 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:29.578320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.579790 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.579795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:29.580867 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:29.580879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:29.581859 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:29.581863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:29.581993 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:29.582022 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 13:43:29.610348 (+ 28326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 13:43:29.610564 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.237.254:38083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
W20260927 13:43:29.610940 7501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51920: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 13:43:29.611994 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7432
I20260927 13:43:29.613158 7493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:29.702819 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7432
2026-09-27T13:43:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (662 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (379 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:30 dist-test-slave-7g3l krb5kdc[7542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:30 dist-test-slave-7g3l krb5kdc[7542](info): set up 2 sockets
Sep 27 13:43:30 dist-test-slave-7g3l krb5kdc[7542](info): commencing operation
krb5kdc: starting...
W20260927 13:43:31.523618 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:31 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:31.700078 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41261
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:31.996088 7555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:31.996332 7555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:31.996505 7555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:32.003299 7555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:32.003396 7555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:32.003499 7555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:32.003612 7555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:32.003693 7555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:32.013042 7555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:32.014176 7555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:32.015401 7555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:32.025862 7563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:32.025944 7560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:32.025913 7561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:32.027115 7555 server_base.cc:1034] running on GCE node
I20260927 13:43:32.027753 7555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:32.028836 7555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:32.030042 7555 hybrid_clock.cc:648] HybridClock initialized: now 1790516612029995 us; error 55 us; skew 500 ppm
I20260927 13:43:32.030340 7555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:32 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516612, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:32.036825 7555 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:32.038026 7555 webserver.cc:460] Webserver started at http://127.1.237.254:45203/ using document root <none> and password file <none>
I20260927 13:43:32.038558 7555 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:32.038650 7555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:32.038892 7555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:32.041605 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "737f03fb40da4f01b6f8e37e37bc5136"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "4c331f9f138101e760a6b579e7cd2bb9"
server_key_iv: "1672efb77ae5bab2e72b522ca697d64e"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.042299 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "737f03fb40da4f01b6f8e37e37bc5136"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "4c331f9f138101e760a6b579e7cd2bb9"
server_key_iv: "1672efb77ae5bab2e72b522ca697d64e"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.046875 7555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 13:43:32.049885 7570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:32.051110 7555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:32.051337 7555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "737f03fb40da4f01b6f8e37e37bc5136"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "4c331f9f138101e760a6b579e7cd2bb9"
server_key_iv: "1672efb77ae5bab2e72b522ca697d64e"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.051512 7555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:32.082958 7555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:32.086833 7555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:32.087091 7555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:32.104205 7555 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:41261
I20260927 13:43:32.104274 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:41261 every 8 connection(s)
I20260927 13:43:32.105445 7555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:32.105934 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7555
I20260927 13:43:32.106127 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:32.106930 1975 external_mini_cluster.cc:1428] Setting key 661935b539ab2bcd4a8c9f53cde70193
I20260927 13:43:32.109503 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:43:32 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:32.121659 7622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136: Bootstrap starting.
I20260927 13:43:32.125463 7622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:32.126668 7622 log.cc:824] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:32.128720 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:32.108804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.1.237.254:41261)
0927 13:43:32.109612 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:32.109627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:32.109678 (+ 51us) server_negotiation.cc:407] Connection header received
0927 13:43:32.111314 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:32.111333 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:32.111685 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:32.112015 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:32.113533 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.114768 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.116467 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.117079 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.117194 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:32.121454 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:32.121465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:32.121480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:32.121526 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:32.125072 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:32.125682 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:32.125690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:32.125697 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:32.125907 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:32.126248 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:32.126253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:32.126256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:32.126888 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:32.127200 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:32.127517 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 13:43:32.127832 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260927 13:43:32.129347 7622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136: No bootstrap required, opened a new log
I20260927 13:43:32.136596 7622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } }
I20260927 13:43:32.136902 7622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:32.137007 7622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737f03fb40da4f01b6f8e37e37bc5136, State: Initialized, Role: FOLLOWER
I20260927 13:43:32.137542 7622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } }
I20260927 13:43:32.137751 7622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:32.137876 7622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:32.138044 7622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:32.140065 7622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } }
I20260927 13:43:32.140478 7622 leader_election.cc:302] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737f03fb40da4f01b6f8e37e37bc5136; no voters:
I20260927 13:43:32.140996 7622 leader_election.cc:288] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:32.141115 7627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:32.141793 7627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [term 1 LEADER]: Becoming Leader. State: Replica: 737f03fb40da4f01b6f8e37e37bc5136, State: Running, Role: LEADER
I20260927 13:43:32.142280 7627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } }
I20260927 13:43:32.142855 7622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:32.147020 7628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737f03fb40da4f01b6f8e37e37bc5136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } } }
I20260927 13:43:32.147475 7628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:32.153541 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737f03fb40da4f01b6f8e37e37bc5136. Latest consensus state: current_term: 1 leader_uuid: "737f03fb40da4f01b6f8e37e37bc5136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737f03fb40da4f01b6f8e37e37bc5136" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41261 } } }
I20260927 13:43:32.155614 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:32.160442 7636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:32.167856 7636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:32.177275 7636 catalog_manager.cc:1347] Generated new cluster ID: 6bf616354465442bade6c74bbdb1958b
I20260927 13:43:32.177414 7636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:32.192024 7636 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:32.193779 7636 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:32.206064 7636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737f03fb40da4f01b6f8e37e37bc5136: Generated new TSK 0
I20260927 13:43:32.207003 7636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:32.253942 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:32.561802 7648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:32.562098 7648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:32.562252 7648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:32.568928 7648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:32.569061 7648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:32.569214 7648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:32.578414 7648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:32.579614 7648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:32.580844 7648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:32.592882 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:32.593024 7653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:32.592959 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:32.594159 7648 server_base.cc:1034] running on GCE node
I20260927 13:43:32.594681 7648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:32.595634 7648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:32.596825 7648 hybrid_clock.cc:648] HybridClock initialized: now 1790516612596791 us; error 35 us; skew 500 ppm
I20260927 13:43:32.597131 7648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:32 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516612, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:32.600178 7648 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:32.601528 7648 webserver.cc:460] Webserver started at http://127.1.237.193:41625/ using document root <none> and password file <none>
I20260927 13:43:32.602150 7648 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:32.602275 7648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:32.602579 7648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:32.605297 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "9c62bbd377d24035a4ca6c9427cd896d"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "8f58b73d0888954927d472be0c501df4"
server_key_iv: "9e780c206a0064c0888dcff4cbf94553"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.606070 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "9c62bbd377d24035a4ca6c9427cd896d"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "8f58b73d0888954927d472be0c501df4"
server_key_iv: "9e780c206a0064c0888dcff4cbf94553"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.610756 7648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 13:43:32.613969 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:32.615147 7648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:32.615358 7648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "9c62bbd377d24035a4ca6c9427cd896d"
format_stamp: "Formatted at 2026-09-27 13:43:32 on dist-test-slave-7g3l"
server_key: "8f58b73d0888954927d472be0c501df4"
server_key_iv: "9e780c206a0064c0888dcff4cbf94553"
server_key_version: "encryptionkey@0"
I20260927 13:43:32.615554 7648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:32.637974 7648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:32.642047 7648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:32.642314 7648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:32.643224 7648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:32.644556 7648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:32.644655 7648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:32.644801 7648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:32.644871 7648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:32.676568 7648 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:36727
I20260927 13:43:32.676641 7775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:36727 every 8 connection(s)
I20260927 13:43:32.677743 7648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:32.682032 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7648
I20260927 13:43:32.682222 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:32.682987 1975 external_mini_cluster.cc:1428] Setting key a5729d1722a2bf630dfe5894267a37de
Sep 27 13:43:32 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516612, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:32.699179 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:32.680091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:49335 (local address 127.1.237.254:41261)
0927 13:43:32.680455 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:32.680460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:32.681501 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 13:43:32.683966 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:32.683972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:32.684060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:32.684172 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:32.685791 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.686449 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.688802 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.689192 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.689281 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:32.692865 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:32.692874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:32.692882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:32.692922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:32.696232 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:32.696845 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:32.696853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:32.696859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:32.697002 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:32.697354 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:32.697361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:32.697368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:32.697728 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:32.697872 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:32.698443 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 13:43:32.698682 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":186,"threads_started":1}
I20260927 13:43:32.699684 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:32.680650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41261 (local address 127.1.237.193:49335)
0927 13:43:32.681352 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:32.681393 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:32.683725 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:32.684384 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:32.684391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:32.684908 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:32.685628 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:32.685657 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.686595 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.686603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:32.688676 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:32.688692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:32.689401 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:32.689410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:32.689701 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:32.690379 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:32.690410 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:32.692652 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:32.696348 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:32.696354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:32.696364 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:32.696717 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:32.697097 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:32.697100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:32.697102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:32.697249 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:32.697838 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:32.697843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:32.698109 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 13:43:32.698301 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 13:43:32.698613 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":291,"threads_started":1}
I20260927 13:43:32.701035 7776 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41261
I20260927 13:43:32.701390 7776 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:32.702237 7776 heartbeater.cc:511] Master 127.1.237.254:41261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:43:32.704523 7587 ts_manager.cc:194] Registered new tserver with Master: 9c62bbd377d24035a4ca6c9427cd896d (127.1.237.193:36727)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:43:32.706223 7587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:49335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:32.727512 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:33.031351 7782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:33.031641 7782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:33.031818 7782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:33.038880 7782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:33.039024 7782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:33.039237 7782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:33.048310 7782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:33.049427 7782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:33.050608 7782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:33.062995 7790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:33.063126 7787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:33.063095 7788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:33.063905 7782 server_base.cc:1034] running on GCE node
I20260927 13:43:33.064411 7782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:33.065196 7782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:33.066378 7782 hybrid_clock.cc:648] HybridClock initialized: now 1790516613066339 us; error 30 us; skew 500 ppm
I20260927 13:43:33.066653 7782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.069584 7782 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:33.070749 7782 webserver.cc:460] Webserver started at http://127.1.237.194:41743/ using document root <none> and password file <none>
I20260927 13:43:33.071375 7782 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:33.071467 7782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:33.071695 7782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:33.074167 7782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "e1ca1992977149d791fc8b8990c27b1d"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "628a0541cb03f0c6567a449dd577583a"
server_key_iv: "836f659fd0ef4b5c44643780a8f5d9fe"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.074815 7782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "e1ca1992977149d791fc8b8990c27b1d"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "628a0541cb03f0c6567a449dd577583a"
server_key_iv: "836f659fd0ef4b5c44643780a8f5d9fe"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.079259 7782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260927 13:43:33.082149 7797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.083411 7782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 13:43:33.083567 7782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "e1ca1992977149d791fc8b8990c27b1d"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "628a0541cb03f0c6567a449dd577583a"
server_key_iv: "836f659fd0ef4b5c44643780a8f5d9fe"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.083752 7782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:33.098061 7782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:33.102075 7782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:33.102339 7782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:33.103241 7782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:33.104580 7782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:33.104676 7782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.104821 7782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:33.104893 7782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.136947 7782 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:38923
I20260927 13:43:33.136997 7909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:38923 every 8 connection(s)
I20260927 13:43:33.138087 7782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:33.143427 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7782
I20260927 13:43:33.143610 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:33.144147 1975 external_mini_cluster.cc:1428] Setting key 48a02f6be129daec7c506eb7ff5d7210
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:33.159789 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.140443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:58389 (local address 127.1.237.254:41261)
0927 13:43:33.140600 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.140604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.141665 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 13:43:33.143481 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.143498 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.143596 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.143720 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.145462 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.146286 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.148740 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.149125 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.149210 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.153043 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.153050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.153057 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.153098 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.156721 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.157359 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.157367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.157374 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.157527 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.157951 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.157965 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.157971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.158335 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.158490 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.159100 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.159345 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:43:33.160508 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.140868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41261 (local address 127.1.237.194:58389)
0927 13:43:33.141522 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:33.141555 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:33.143197 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:33.144004 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:33.144016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:33.144677 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:33.145307 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:33.145330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.146436 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.146441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:33.148616 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:33.148632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.149326 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.149334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:33.149642 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.150338 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:33.150369 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:33.152837 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:33.156853 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:33.156859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:33.156868 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:33.157231 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:33.157630 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:33.157633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:33.157635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:33.157818 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:33.158450 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:33.158456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:33.158720 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 13:43:33.158921 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 13:43:33.159337 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":269,"threads_started":1}
I20260927 13:43:33.161890 7910 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41261
I20260927 13:43:33.162159 7910 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:33.162799 7910 heartbeater.cc:511] Master 127.1.237.254:41261 requested a full tablet report, sending...
I20260927 13:43:33.164140 7587 ts_manager.cc:194] Registered new tserver with Master: e1ca1992977149d791fc8b8990c27b1d (127.1.237.194:38923)
I20260927 13:43:33.164880 7587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:58389
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:33.190240 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:33.497013 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:33.497315 7915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:33.497466 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:33.504236 7915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:33.504366 7915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:33.504528 7915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:33.513355 7915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:41261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:33.514483 7915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:33.515760 7915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:33.528038 7921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:33.528065 7923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:33.528143 7920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:33.529091 7915 server_base.cc:1034] running on GCE node
I20260927 13:43:33.529637 7915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:33.530480 7915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:33.531711 7915 hybrid_clock.cc:648] HybridClock initialized: now 1790516613531669 us; error 53 us; skew 500 ppm
I20260927 13:43:33.532039 7915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.535017 7915 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:33.536310 7915 webserver.cc:460] Webserver started at http://127.1.237.195:38153/ using document root <none> and password file <none>
I20260927 13:43:33.536913 7915 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:33.537007 7915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:33.537242 7915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:33.539932 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "9ca3e3d3f3624c6395f1f8c8a5a0cc13"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "95c4062aa069b62abcfb6d5074ce97d9"
server_key_iv: "63001ef418d07582bfad4c75d48e19f7"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.540603 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "9ca3e3d3f3624c6395f1f8c8a5a0cc13"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "95c4062aa069b62abcfb6d5074ce97d9"
server_key_iv: "63001ef418d07582bfad4c75d48e19f7"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.545400 7915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:43:33.548451 7930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.549642 7915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:33.549793 7915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "9ca3e3d3f3624c6395f1f8c8a5a0cc13"
format_stamp: "Formatted at 2026-09-27 13:43:33 on dist-test-slave-7g3l"
server_key: "95c4062aa069b62abcfb6d5074ce97d9"
server_key_iv: "63001ef418d07582bfad4c75d48e19f7"
server_key_version: "encryptionkey@0"
I20260927 13:43:33.549970 7915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:33.565245 7915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:33.568562 7915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:33.568817 7915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:33.569658 7915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:33.570812 7915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:33.570906 7915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.571051 7915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:33.571118 7915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:33.601303 7915 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:40887
I20260927 13:43:33.601353 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:40887 every 8 connection(s)
I20260927 13:43:33.602523 7915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:33.605456 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 7915
I20260927 13:43:33.605638 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:33.606244 1975 external_mini_cluster.cc:1428] Setting key bfee2c008a439c0096d1477a5ee4bdf3
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:33.621459 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.604935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:51303 (local address 127.1.237.254:41261)
0927 13:43:33.605116 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.605120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.606593 (+ 1473us) server_negotiation.cc:407] Connection header received
0927 13:43:33.608865 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.608872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.608954 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.609030 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.610369 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.611074 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.612736 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.613114 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.613202 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.616159 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.616163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.616166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.616205 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.618385 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.619080 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.619084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.619086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.619255 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.619708 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.619712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.619715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.620086 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.620217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.620988 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.621233 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:43:33.622315 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.605588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41261 (local address 127.1.237.195:51303)
0927 13:43:33.606431 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:33.606476 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:33.608577 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:33.609259 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:33.609266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:33.609716 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:33.610223 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:33.610238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.611259 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.611266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:33.612621 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:33.612633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.613280 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.613284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:33.613461 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.614025 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:33.614044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:33.616004 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:33.618501 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:33.618507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:33.618553 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:33.618947 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:33.619372 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:33.619375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:33.619377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:33.619573 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:33.620207 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:33.620212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:33.620596 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 13:43:33.620884 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 13:43:33.621296 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":327,"threads_started":1}
I20260927 13:43:33.623587 8043 heartbeater.cc:347] Connected to a master server at 127.1.237.254:41261
I20260927 13:43:33.623859 8043 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:33.624519 8043 heartbeater.cc:511] Master 127.1.237.254:41261 requested a full tablet report, sending...
I20260927 13:43:33.625773 7587 ts_manager.cc:194] Registered new tserver with Master: 9ca3e3d3f3624c6395f1f8c8a5a0cc13 (127.1.237.195:40887)
I20260927 13:43:33.626487 7587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:51303
I20260927 13:43:33.632795 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/test-admin.keytab.
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.657483 1975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 13:43:33.662119 8050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:33.659948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.1.237.193:36727)
0927 13:43:33.660586 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.660592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.660657 (+ 65us) server_negotiation.cc:407] Connection header received
0927 13:43:33.660710 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.660717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.660910 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.661026 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.661800 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51564: BlockingRecv error: recv got EOF from 127.0.0.1:51564 (error 108)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":288,"threads_started":1}
W20260927 13:43:33.662191 1975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.664143 1975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.665510 1975 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:33.679868 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.666892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.1.237.193:36727)
0927 13:43:33.667063 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.667069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.667307 (+ 238us) server_negotiation.cc:407] Connection header received
0927 13:43:33.667406 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.667410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.667458 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.667539 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.668429 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.669388 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.670495 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.671000 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.671089 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.674210 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.674220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.674232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.674259 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.677404 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.677946 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.677955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.677958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.678080 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.678386 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.678393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.678397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.678797 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.679099 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.679301 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.679521 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 13:43:33.684810 8050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:33.683382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.1.237.193:36727)
0927 13:43:33.683545 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.683552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.683849 (+ 297us) server_negotiation.cc:407] Connection header received
0927 13:43:33.683951 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.683954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.683998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.684066 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.684620 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51594: BlockingRecv error: recv got EOF from 127.0.0.1:51594 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 13:43:33.684968 1975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.686257 1975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.687633 1975 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:33.699342 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.688732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.1.237.193:36727)
0927 13:43:33.688899 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.688904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.689062 (+ 158us) server_negotiation.cc:407] Connection header received
0927 13:43:33.689143 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.689146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.689189 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.689258 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.690076 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.690725 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.691866 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.692271 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.692385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.695410 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.695414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.695417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.695450 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.697537 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.698025 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.698028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.698030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.698122 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.698388 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.698391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.698393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.698596 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.698751 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.698925 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.699076 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 13:43:33.703019 8050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:33.701458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.1.237.193:36727)
0927 13:43:33.701649 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.701656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.701878 (+ 222us) server_negotiation.cc:407] Connection header received
0927 13:43:33.702038 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.702043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.702115 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.702207 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.702804 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51616: BlockingRecv error: recv got EOF from 127.0.0.1:51616 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 13:43:33.703222 1975 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790516567150638-1975-0/krb5kdc/krb5cc' of type FILE
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.704564 1975 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:33.705806 1975 init.cc:295] Successfully reacquired a new kerberos TGT
I20260927 13:43:33.709556 7776 heartbeater.cc:503] Master 127.1.237.254:41261 was elected leader, sending a full tablet report...
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:33.718740 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.707033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.1.237.193:36727)
0927 13:43:33.707217 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.707223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.707399 (+ 176us) server_negotiation.cc:407] Connection header received
0927 13:43:33.707570 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.707575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.707629 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.707719 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.708606 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.709274 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.710638 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.711050 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.711211 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.714196 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.714200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.714203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.714228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.716467 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.717025 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.717032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.717040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.717166 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.717496 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.717501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.717505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.717743 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.718025 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.718294 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.718484 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:43:33.720052 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7648
I20260927 13:43:33.730377 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.720635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.1.237.193:36727)
0927 13:43:33.720751 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.720755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.721122 (+ 367us) server_negotiation.cc:407] Connection header received
0927 13:43:33.721234 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.721237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.721280 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.721349 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.722133 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.722921 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.724063 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.724436 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.724522 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.726181 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.726185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.726188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.726212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.728552 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.729060 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.729064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.729066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.729179 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.729446 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.729449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.729451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.729651 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.729814 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.730040 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.730171 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 13:43:33.730891 7771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:33.809897 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7648
I20260927 13:43:33.825212 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7782
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:33.839390 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.826024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.1.237.194:38923)
0927 13:43:33.826485 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.826491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.826522 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:43:33.826575 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.826581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.826781 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.826898 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.827780 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.828740 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.829886 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.830414 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.830509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.834086 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.834096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.834108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.834135 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.836900 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.837460 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.837466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.837468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.837584 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.837862 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.837865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.837867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.838208 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.838438 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.838687 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.838937 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260927 13:43:33.840826 7905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:33.921267 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7782
I20260927 13:43:33.936652 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7915
Sep 27 13:43:33 dist-test-slave-7g3l krb5kdc[7542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:33.951354 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:33.937614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.1.237.195:40887)
0927 13:43:33.938270 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:33.938277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:33.938303 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:43:33.938362 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:33.938367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:33.938577 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:33.938699 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:33.939719 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.940662 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.941825 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:33.942337 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:33.942471 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:33.946429 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:33.946439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:33.946452 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:33.946481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:33.948965 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.949605 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.949611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.949614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.949732 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:33.950047 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:33.950051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:33.950053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:33.950356 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:33.950566 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:33.950736 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 13:43:33.950980 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260927 13:43:33.952570 8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:34.032373 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7915
I20260927 13:43:34.048028 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 7555
I20260927 13:43:34.049192 7617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:34.145733 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 7555
2026-09-27T13:43:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3678 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:34 dist-test-slave-7g3l krb5kdc[8070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:34 dist-test-slave-7g3l krb5kdc[8070](info): set up 2 sockets
Sep 27 13:43:34 dist-test-slave-7g3l krb5kdc[8070](info): commencing operation
krb5kdc: starting...
W20260927 13:43:35.203718 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:35 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab.
Entry for principal oryx/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab.
I20260927 13:43:35.385788 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34881
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab
--principal=oryx/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:35.690517 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:35.690814 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:35.690975 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:35.697799 8083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:35.697924 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:35.698030 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:35.698130 8083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:35.698220 8083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:35.707470 8083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab
--principal=oryx/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:35.708673 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:35.709905 8083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:35.720839 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:35.720929 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:35.720885 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:35.721786 8083 server_base.cc:1034] running on GCE node
I20260927 13:43:35.722425 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:35.723658 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:35.724881 8083 hybrid_clock.cc:648] HybridClock initialized: now 1790516615724808 us; error 37 us; skew 500 ppm
I20260927 13:43:35.725199 8083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:35 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:35.728039 8083 init.cc:377] Logged in from keytab as oryx/127.1.237.254@KRBTEST.COM (short username oryx)
I20260927 13:43:35.729461 8083 webserver.cc:460] Webserver started at http://127.1.237.254:33903/ using document root <none> and password file <none>
I20260927 13:43:35.730072 8083 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:35.730190 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:35.730473 8083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:35.733258 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "578daa7b1b23425589168c5d6c66bf0e"
format_stamp: "Formatted at 2026-09-27 13:43:35 on dist-test-slave-7g3l"
server_key: "a62a7e4daa07803fa38f2641a0be82d3"
server_key_iv: "ce41c4a8ac6692e4d89627602970d032"
server_key_version: "encryptionkey@0"
I20260927 13:43:35.733991 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "578daa7b1b23425589168c5d6c66bf0e"
format_stamp: "Formatted at 2026-09-27 13:43:35 on dist-test-slave-7g3l"
server_key: "a62a7e4daa07803fa38f2641a0be82d3"
server_key_iv: "ce41c4a8ac6692e4d89627602970d032"
server_key_version: "encryptionkey@0"
I20260927 13:43:35.738633 8083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 13:43:35.741873 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:35.743120 8083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:35.743304 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "578daa7b1b23425589168c5d6c66bf0e"
format_stamp: "Formatted at 2026-09-27 13:43:35 on dist-test-slave-7g3l"
server_key: "a62a7e4daa07803fa38f2641a0be82d3"
server_key_iv: "ce41c4a8ac6692e4d89627602970d032"
server_key_version: "encryptionkey@0"
I20260927 13:43:35.743486 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:35.795310 8083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:35.799330 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:35.799593 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:35.817504 8083 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:34881
I20260927 13:43:35.817553 8149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:34881 every 8 connection(s)
I20260927 13:43:35.818748 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:35.820586 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8083
I20260927 13:43:35.820783 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:35.821381 1975 external_mini_cluster.cc:1428] Setting key 8c005467802daa1589a50c6b8a94a8f9
I20260927 13:43:35.822439 8150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:43:35 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:35.834940 8150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Bootstrap starting.
I20260927 13:43:35.838728 8150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:35.840015 8150 log.cc:824] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:35.843050 8150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: No bootstrap required, opened a new log
I20260927 13:43:35.843297 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:35.823330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.1.237.254:34881)
0927 13:43:35.824051 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:35.824065 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:35.824119 (+ 54us) server_negotiation.cc:407] Connection header received
0927 13:43:35.825704 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:35.825729 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:35.826219 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:35.826636 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:35.827817 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:35.829702 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:35.830819 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:35.831818 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:35.832007 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:35.835566 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:35.835575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:35.835592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:35.835630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:35.839310 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:35.840039 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:35.840047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:35.840054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:35.840247 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:35.840650 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:35.840657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:35.840660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:35.841101 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:35.841405 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:35.841701 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 13:43:35.842086 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":294,"threads_started":1}
I20260927 13:43:35.850770 8150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } }
I20260927 13:43:35.851064 8150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:35.851130 8150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578daa7b1b23425589168c5d6c66bf0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:35.851677 8150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } }
I20260927 13:43:35.851858 8150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:35.851964 8150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:35.852078 8150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:35.854048 8150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } }
I20260927 13:43:35.854440 8150 leader_election.cc:302] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 578daa7b1b23425589168c5d6c66bf0e; no voters:
I20260927 13:43:35.854874 8150 leader_election.cc:288] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:35.855043 8155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:35.855686 8155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 LEADER]: Becoming Leader. State: Replica: 578daa7b1b23425589168c5d6c66bf0e, State: Running, Role: LEADER
I20260927 13:43:35.856134 8155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } }
I20260927 13:43:35.856739 8150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:35.858990 8157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 578daa7b1b23425589168c5d6c66bf0e. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } }
I20260927 13:43:35.859457 8157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:35.859071 8156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } }
I20260927 13:43:35.860291 8156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:35.861690 8163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:35.866214 8163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:35.873430 8163 catalog_manager.cc:1347] Generated new cluster ID: 5a274b1f63aa4803a437882c40afd6df
I20260927 13:43:35.873538 8163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:35.898092 8163 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:35.899333 8163 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:35.906641 8163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Generated new TSK 0
I20260927 13:43:35.907562 8163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
I20260927 13:43:35.962421 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:36.269129 8176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:36.269359 8176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:36.269534 8176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:36.277828 8176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:36.277942 8176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:36.278127 8176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:36.287235 8176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:36.288417 8176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:36.289645 8176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:36.302634 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:36.302651 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:36.302899 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:36.303467 8176 server_base.cc:1034] running on GCE node
I20260927 13:43:36.304071 8176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:36.305038 8176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:36.306300 8176 hybrid_clock.cc:648] HybridClock initialized: now 1790516616306221 us; error 69 us; skew 500 ppm
I20260927 13:43:36.306641 8176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:36 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:36.309791 8176 init.cc:377] Logged in from keytab as oryx/127.1.237.193@KRBTEST.COM (short username oryx)
I20260927 13:43:36.311339 8176 webserver.cc:460] Webserver started at http://127.1.237.193:39927/ using document root <none> and password file <none>
I20260927 13:43:36.312033 8176 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:36.312175 8176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:36.312477 8176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:36.315357 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "a1526686335d49e7ba21bd9cb437eb1f"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "47eea133a4cd1672a3dc9e7eb8d81ceb"
server_key_iv: "4f246399710d7841095d60aeebc7361f"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.316092 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "a1526686335d49e7ba21bd9cb437eb1f"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "47eea133a4cd1672a3dc9e7eb8d81ceb"
server_key_iv: "4f246399710d7841095d60aeebc7361f"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.321085 8176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:43:36.324324 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.325564 8176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:43:36.325726 8176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a1526686335d49e7ba21bd9cb437eb1f"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "47eea133a4cd1672a3dc9e7eb8d81ceb"
server_key_iv: "4f246399710d7841095d60aeebc7361f"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.325917 8176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:36.339054 8176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:36.342664 8176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:36.342933 8176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:36.343799 8176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:36.345006 8176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:36.345103 8176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.345242 8176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:36.345314 8176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.377487 8176 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:33031
I20260927 13:43:36.377565 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:33031 every 8 connection(s)
I20260927 13:43:36.378660 8176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:36.379232 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8176
I20260927 13:43:36.379448 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:36.380234 1975 external_mini_cluster.cc:1428] Setting key 6dc48b198ee73c5889f6b45492f236c1
Sep 27 13:43:36 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:36.398901 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:36.381236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52423 (local address 127.1.237.254:34881)
0927 13:43:36.381681 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:36.381686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:36.382597 (+ 911us) server_negotiation.cc:407] Connection header received
0927 13:43:36.384320 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:36.384326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:36.384398 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:36.384519 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:36.385856 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.386563 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.388134 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.388783 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.388920 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:36.392064 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:36.392073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:36.392081 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:36.392123 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:36.395556 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:36.396531 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:36.396535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:36.396538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:36.396723 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:36.397251 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:36.397255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:36.397257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:36.397517 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:36.397632 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:36.398523 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 13:43:36.398671 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":220,"threads_started":1}
I20260927 13:43:36.400278 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:36.381773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:52423)
0927 13:43:36.382455 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:36.382501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:36.384061 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:36.384738 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:36.384746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:36.385196 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:36.385701 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:36.385716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.386711 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.386718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:36.387992 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:36.388003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.388986 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.388991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:36.389174 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:36.389797 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:36.389817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:36.391819 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:36.395721 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:36.395732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:36.395750 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:36.396371 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:36.396880 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:36.396887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:36.396892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:36.397135 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:36.397662 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:36.397672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:36.398144 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 13:43:36.398460 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 13:43:36.398951 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":291,"threads_started":1}
WARNING: no policy specified for oryx/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:43:36.401747 8304 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:36.402010 8304 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:36.402698 8304 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.194@KRBTEST.COM" created.
I20260927 13:43:36.404794 8115 ts_manager.cc:194] Registered new tserver with Master: a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031)
I20260927 13:43:36.406150 8115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:52423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
I20260927 13:43:36.424347 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:36.723276 8310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:36.723538 8310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:36.723629 8310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:36.730297 8310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:36.730386 8310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:36.730495 8310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:36.739450 8310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:36.740554 8310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:36.741760 8310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:36.753890 8316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:36.753904 8318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:36.753901 8315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:36.754650 8310 server_base.cc:1034] running on GCE node
I20260927 13:43:36.755208 8310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:36.755985 8310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:36.757167 8310 hybrid_clock.cc:648] HybridClock initialized: now 1790516616757132 us; error 26 us; skew 500 ppm
I20260927 13:43:36.757447 8310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:36 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:36.760413 8310 init.cc:377] Logged in from keytab as oryx/127.1.237.194@KRBTEST.COM (short username oryx)
I20260927 13:43:36.761725 8310 webserver.cc:460] Webserver started at http://127.1.237.194:37345/ using document root <none> and password file <none>
I20260927 13:43:36.762336 8310 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:36.762425 8310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:36.762655 8310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:36.765442 8310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "867bbe567f7f431b88e06f660bfc65a7"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "29706c833a7fdeaa8804efc2616e4e5a"
server_key_iv: "4fe2224a72de100337a84de73b062967"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.766165 8310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "867bbe567f7f431b88e06f660bfc65a7"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "29706c833a7fdeaa8804efc2616e4e5a"
server_key_iv: "4fe2224a72de100337a84de73b062967"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.770756 8310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:43:36.773955 8325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.775317 8310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:36.775492 8310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "867bbe567f7f431b88e06f660bfc65a7"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "29706c833a7fdeaa8804efc2616e4e5a"
server_key_iv: "4fe2224a72de100337a84de73b062967"
server_key_version: "encryptionkey@0"
I20260927 13:43:36.775671 8310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:36.789362 8310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:36.793311 8310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:36.793581 8310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:36.794447 8310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:36.795715 8310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:36.795810 8310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.795960 8310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:36.796036 8310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:36.828473 8310 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:43445
I20260927 13:43:36.828490 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:43445 every 8 connection(s)
I20260927 13:43:36.829855 8310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:36.839509 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8310
I20260927 13:43:36.839670 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:36.840101 1975 external_mini_cluster.cc:1428] Setting key 035a46a91055f480a22ec5e84b446470
Sep 27 13:43:36 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:36.848743 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:36.832400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:37979 (local address 127.1.237.254:34881)
0927 13:43:36.832569 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:36.832573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:36.833784 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 13:43:36.835556 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:36.835561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:36.835644 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:36.835751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:36.837099 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.837791 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.839268 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.839745 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.839892 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:36.843405 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:36.843417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:36.843420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:36.843448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:36.845826 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:36.846471 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:36.846488 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:36.846490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:36.846668 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:36.847057 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:36.847062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:36.847065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:36.847363 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:36.847509 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:36.848192 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 13:43:36.848392 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:43:36.849592 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:36.832956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:37979)
0927 13:43:36.833639 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:36.833678 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:36.835311 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:36.835975 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:36.835988 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:36.836439 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:36.836935 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:36.836950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.837937 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.837949 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:36.839108 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:36.839118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:36.839964 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:36.839971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:36.840248 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:36.840903 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:36.840929 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:36.843223 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:36.845955 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:36.845963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:36.845976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:36.846354 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:36.846773 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:36.846777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:36.846781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:36.846945 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:36.847475 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:36.847482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:36.847860 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 13:43:36.848104 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 13:43:36.848550 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260927 13:43:36.850852 8438 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:36.851127 8438 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:36.851783 8438 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
I20260927 13:43:36.853137 8115 ts_manager.cc:194] Registered new tserver with Master: 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:36.854342 8115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:37979
WARNING: no policy specified for oryx/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
I20260927 13:43:36.880497 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:37.184320 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:37.184548 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:37.184631 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:37.191332 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:37.191413 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:37.191521 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:37.200090 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:37.201047 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:37.202167 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:37.213897 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:37.214012 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:37.214012 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:37.214745 8443 server_base.cc:1034] running on GCE node
I20260927 13:43:37.215268 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:37.216014 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:37.217204 8443 hybrid_clock.cc:648] HybridClock initialized: now 1790516617217171 us; error 23 us; skew 500 ppm
I20260927 13:43:37.217473 8443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516617, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:37.225097 8443 init.cc:377] Logged in from keytab as oryx/127.1.237.195@KRBTEST.COM (short username oryx)
I20260927 13:43:37.226375 8443 webserver.cc:460] Webserver started at http://127.1.237.195:42121/ using document root <none> and password file <none>
I20260927 13:43:37.227005 8443 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:37.227118 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:37.227427 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:37.230306 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "3c4f17f14f644f47bd427309a83ad90b"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "ea8a740e9d7202ac91b985d7581c3049"
server_key_iv: "c17111998139610becf0436618603f69"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.231047 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "3c4f17f14f644f47bd427309a83ad90b"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "ea8a740e9d7202ac91b985d7581c3049"
server_key_iv: "c17111998139610becf0436618603f69"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.235617 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:43:37.238535 8458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:37.239797 8443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:37.239959 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "3c4f17f14f644f47bd427309a83ad90b"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "ea8a740e9d7202ac91b985d7581c3049"
server_key_iv: "c17111998139610becf0436618603f69"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.240163 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:37.258015 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:37.261472 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:37.261706 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:37.262542 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:37.263908 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:37.264010 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:37.264145 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:37.264215 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:37.296876 8443 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:39839
I20260927 13:43:37.296916 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:39839 every 8 connection(s)
I20260927 13:43:37.298036 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:37.305545 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8443
I20260927 13:43:37.305706 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:37.306136 1975 external_mini_cluster.cc:1428] Setting key c0a05e24b7582886bb93affd72361a63
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516617, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:37.316504 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.300438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45725 (local address 127.1.237.254:34881)
0927 13:43:37.300605 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.300609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.301670 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 13:43:37.303537 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.303542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.303638 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.303771 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.305175 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.306070 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.307920 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.308418 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.308530 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.311870 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.311875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.311878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.311918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.314206 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.314797 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.314800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.314803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.314900 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.315274 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.315277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.315279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.315501 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.315608 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.316128 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.316279 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:43:37.317375 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.300867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:45725)
0927 13:43:37.301519 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.301557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.303270 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.304013 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.304021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.304491 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.305011 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.305028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.306233 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.306237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.307771 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.307789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.308666 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.308674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.308969 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.309633 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.309658 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.311700 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.314316 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.314322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.314332 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.314698 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.314987 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.314990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.314992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.315145 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.315599 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.315604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.315853 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.316053 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.316382 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260927 13:43:37.318548 8571 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:37.318768 8571 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:37.319409 8571 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
I20260927 13:43:37.320807 8114 ts_manager.cc:194] Registered new tserver with Master: 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:37.321622 8114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:45725
I20260927 13:43:37.332739 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.237.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.253.keytab.
Entry for principal oryx/127.1.237.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.253.keytab.
I20260927 13:43:37.374346 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:38091
--webserver_interface=127.1.237.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.253.keytab
--principal=oryx/127.1.237.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34881,127.1.237.253:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:43:37.409139 8304 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
W20260927 13:43:37.679543 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:37.679796 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:37.679898 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:37.686565 8576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:37.686647 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:37.686690 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:37.686745 8576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:37.686779 8576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:37.695732 8576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34881,127.1.237.253:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.253.keytab
--principal=oryx/127.1.237.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.237.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
W20260927 13:43:37.695993 8576 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.237.254:34881,127.1.237.253:38091'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 13:43:37.696837 8576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:37.697953 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:37.709100 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:37.709056 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:37.709033 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:37.710028 8576 server_base.cc:1034] running on GCE node
I20260927 13:43:37.710750 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:37.712100 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:37.713339 8576 hybrid_clock.cc:648] HybridClock initialized: now 1790516617713277 us; error 36 us; skew 500 ppm
I20260927 13:43:37.713653 8576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516617, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:37.716610 8576 init.cc:377] Logged in from keytab as oryx/127.1.237.253@KRBTEST.COM (short username oryx)
I20260927 13:43:37.717875 8576 webserver.cc:460] Webserver started at http://127.1.237.253:42229/ using document root <none> and password file <none>
I20260927 13:43:37.718631 8576 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:37.718748 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:37.719059 8576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:37.721691 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/instance:
uuid: "041a10c1b226426c96f64572df2f027f"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "620a233e4504ca823ba586e085d3591f"
server_key_iv: "377dbe03d89f11563f14732fb431e46a"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.722419 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal/instance:
uuid: "041a10c1b226426c96f64572df2f027f"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "620a233e4504ca823ba586e085d3591f"
server_key_iv: "377dbe03d89f11563f14732fb431e46a"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.726936 8576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:43:37.730018 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:37.731284 8576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:43:37.731444 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "041a10c1b226426c96f64572df2f027f"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "620a233e4504ca823ba586e085d3591f"
server_key_iv: "377dbe03d89f11563f14732fb431e46a"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.731643 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:37.760003 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:37.764246 8576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:37.764505 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:37.781960 8576 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.253:38091
I20260927 13:43:37.781977 8642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.253:38091 every 8 connection(s)
I20260927 13:43:37.783260 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
I20260927 13:43:37.786733 8643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:37.790207 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8576
I20260927 13:43:37.794010 8643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516617, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.253@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:37.811486 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.797186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.1.237.254:34881)
0927 13:43:37.797332 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.797336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.798074 (+ 738us) server_negotiation.cc:407] Connection header received
0927 13:43:37.798268 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.798272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.798327 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.798422 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.799679 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.800506 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.802315 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.802689 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.802775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.806416 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.806420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.806423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.806449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.808782 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.809513 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.809519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.809523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.809658 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.810034 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.810041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.810045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.810324 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.810475 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.810951 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.811145 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:37.812407 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.792383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.1.237.253:38091)
0927 13:43:37.793113 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.793124 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.793162 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:43:37.794640 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.794660 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.795017 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.795391 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.796982 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.798711 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.800234 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.801032 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.801159 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.805874 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.805884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.805897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.805935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.808887 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.809525 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.809533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.809537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.809688 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.810049 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.810054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.810057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.810478 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.810785 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.811090 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.811533 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":529,"thread_start_us":358,"threads_started":1}
I20260927 13:43:37.812451 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.797293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52414)
0927 13:43:37.797942 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.797976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.798137 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.798470 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.798477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.799238 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.799536 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.799548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.800646 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.800650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.802168 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.802186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.802918 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.802930 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.803141 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.803750 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.803776 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.806248 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.808898 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.808906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.808918 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.809373 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.809787 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.809790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.809792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.809922 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.810453 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.810460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.810633 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.810878 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.811354 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":332,"threads_started":1}
I20260927 13:43:37.814049 8643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516617, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.253@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:37.825548 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.814676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59174)
0927 13:43:37.814861 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.814880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.814967 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.815288 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.815293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.815599 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.815704 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.815711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.816598 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.816603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.817508 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.817518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.818131 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.818136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.818321 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.818813 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.818827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.820570 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.823730 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.823734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.823736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.824060 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.824399 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.824402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.824404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.824510 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.825012 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.825015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.825104 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.825199 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.825367 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260927 13:43:37.825776 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.814825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.1.237.253:38091)
0927 13:43:37.814986 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.815000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.815015 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:43:37.815057 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.815061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.815121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.815261 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.815823 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.816470 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.817631 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.818003 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.818126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.820725 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.820731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.820736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.820768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.823615 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.824162 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.824167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.824171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.824303 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.824608 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.824612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.824616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.824896 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.825034 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.825309 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.825476 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":51}
I20260927 13:43:37.830569 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Bootstrap starting.
I20260927 13:43:37.833942 8643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:37.834824 8643 log.cc:824] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:37.837257 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: No bootstrap required, opened a new log
I20260927 13:43:37.843782 8643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } }
I20260927 13:43:37.844028 8643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:37.844134 8643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041a10c1b226426c96f64572df2f027f, State: Initialized, Role: FOLLOWER
I20260927 13:43:37.844627 8643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } }
I20260927 13:43:37.845314 8649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } } }
I20260927 13:43:37.845615 8649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:37.846182 8643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:43:37.854168 8660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:43:37.854285 8660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:43:37.855885 8576 master_runner.cc:428] Detected that this master 041a10c1b226426c96f64572df2f027f is joining an existing cluster
I20260927 13:43:37.855964 8576 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.237.253:38091
I20260927 13:43:37.856735 8438 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
I20260927 13:43:37.874312 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.860511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.1.237.254:34881)
0927 13:43:37.860707 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.860715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.861281 (+ 566us) server_negotiation.cc:407] Connection header received
0927 13:43:37.861455 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.861460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.861526 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.861633 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.862599 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.863492 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.864787 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.865182 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.865320 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.867369 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.867375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.867380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.867417 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.871059 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.871843 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.871850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.871856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.872016 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.872458 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.872465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.872471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.872810 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.872932 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.873584 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.873853 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:43:37.875537 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.859808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52418)
0927 13:43:37.861177 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.861201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.861341 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.861674 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.861680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.862196 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.862471 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.862479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.863616 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.863623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.864652 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.864663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.865357 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.865364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.865519 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.866155 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.866169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.867128 (+ 959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.871229 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.871240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.871246 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.871713 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.872160 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.872167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.872173 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.872337 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.873048 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.873055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.873270 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.874592 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.874975 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"threads_started":1}
I20260927 13:43:37.889406 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.860154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.1.237.253:38091)
0927 13:43:37.860368 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.860377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.875338 (+ 14961us) server_negotiation.cc:407] Connection header received
0927 13:43:37.875713 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.875718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.875774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.875838 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.877264 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.878286 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.880173 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.880822 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.880916 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.883088 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.883095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.883100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.883143 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.886615 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.887409 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.887414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.887417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.887549 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.887947 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.887957 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.887961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.888303 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.888443 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.888880 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.889086 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:43:37.890626 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.859849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59190)
0927 13:43:37.875252 (+ 15403us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.875270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.875595 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.876166 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.876170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.876812 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.877095 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.877112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.878493 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.878506 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.880020 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.880037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.881175 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.881182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.881345 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.881872 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.881898 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.882894 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.886743 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.886748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.886751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.887295 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.887680 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.887684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.887688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.887833 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.888514 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.888518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.888641 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.889931 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.890176 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15253,"thread_start_us":1032,"threads_started":1}
WARNING: no policy specified for oryx/127.1.237.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.252@KRBTEST.COM" created.
Sep 27 13:43:37 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.254@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:37.909812 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.882549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59206)
0927 13:43:37.894457 (+ 11908us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.894521 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.894863 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.895386 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.895399 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.896041 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.896326 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.896335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.897412 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.897420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.898379 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.898389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.899300 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.899309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.899450 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.900033 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.900049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.901957 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.906808 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.906821 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.906828 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.907434 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.907852 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.907857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.907860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.908038 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.908691 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.908699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.908887 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.909087 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.909392 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11710,"thread_start_us":11571,"threads_started":1}
I20260927 13:43:37.910219 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.882096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.1.237.253:38091)
0927 13:43:37.889899 (+ 7803us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.889904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.894655 (+ 4751us) server_negotiation.cc:407] Connection header received
0927 13:43:37.895100 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.895108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.895240 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.895346 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.896589 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.897284 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.898517 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.898972 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.899091 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.902140 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.902144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.902147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.902172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.906671 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.907559 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.907567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.907574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.907725 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.908163 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.908171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.908177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.908556 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.908698 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.909217 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.909453 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7704,"thread_start_us":13156,"threads_started":1}
I20260927 13:43:37.912386 8114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.237.253:38091: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "041a10c1b226426c96f64572df2f027f"
member_type: NON_VOTER
last_known_addr {
host: "127.1.237.253"
port: 38091
}
attrs {
promote: true
}
}
dest_uuid: "578daa7b1b23425589168c5d6c66bf0e"
cas_config_opid_index: -1
I20260927 13:43:37.913547 8114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } }
I20260927 13:43:37.916621 8674 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 041a10c1b226426c96f64572df2f027f (127.1.237.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } }
I20260927 13:43:37.917693 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:37.920670 8674 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.237.253:38091
I20260927 13:43:37.921502 8618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 13:43:37.921295 8673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } }
I20260927 13:43:37.921607 8673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:37.921592 8576 master.cc:537] Master@127.1.237.253:38091 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.252.keytab.
Entry for principal oryx/127.1.237.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.252.keytab.
I20260927 13:43:37.924362 8673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:37.925030 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:41075
--webserver_interface=127.1.237.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.252.keytab
--principal=oryx/127.1.237.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:37.927124 8100 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 13:43:37.935048 8576 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:37.935562 8576 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:37.935760 8576 tablet_replica.cc:330] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: stopping tablet replica
W20260927 13:43:37.936307 8100 consensus_peers.cc:487] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e -> Peer 041a10c1b226426c96f64572df2f027f (127.1.237.253:38091): Couldn't send request to peer 041a10c1b226426c96f64572df2f027f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 13:43:37.941409 8576 master.cc:552] Master@127.1.237.253:38091 shutdown complete.
I20260927 13:43:37.941521 8576 master_runner.cc:305] Clearing existing system tablet
I20260927 13:43:37.941898 8576 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:37.942004 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:37.944329 8576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:37.945785 8679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:37.946095 8576 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:37.946372 8576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 13:43:37.946513 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "041a10c1b226426c96f64572df2f027f"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "620a233e4504ca823ba586e085d3591f"
server_key_iv: "377dbe03d89f11563f14732fb431e46a"
server_key_version: "encryptionkey@0"
I20260927 13:43:37.950321 8576 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:37.954757 8576 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:43:37.954932 8576 log.cc:1196] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 13:43:37.955364 8576 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Deleting consensus metadata
I20260927 13:43:37.955618 8576 master_runner.cc:315] Copying system tablet from 127.1.237.254:34881
I20260927 13:43:37.958014 8576 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: tablet copy: Beginning tablet copy session from remote peer at address 127.1.237.254:34881
I20260927 13:43:37.969761 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.958662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.1.237.254:34881)
0927 13:43:37.958834 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:37.958844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:37.959339 (+ 495us) server_negotiation.cc:407] Connection header received
0927 13:43:37.959528 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:37.959533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:37.959597 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:37.959685 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:37.960419 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.961090 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.962367 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.962776 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.962929 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.964724 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:37.964748 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:37.964752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:37.964789 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:37.967459 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.968014 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.968021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.968027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.968145 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:37.968473 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:37.968479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:37.968483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:37.968723 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:37.968863 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:37.969217 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 13:43:37.969403 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:43:37.969825 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:37.958653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52434)
0927 13:43:37.959233 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:37.959254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:37.959398 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:37.959704 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:37.959708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:37.960089 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:37.960280 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.960288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.961218 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.961224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.962240 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:37.962251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:37.962931 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:37.962936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:37.963113 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:37.963808 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:37.963825 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:37.964544 (+ 719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:37.967579 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.967585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.967593 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.967904 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.968247 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:37.968252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:37.968256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:37.968377 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:37.968830 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:37.968835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:37.968980 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 13:43:37.969133 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 13:43:37.969399 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":311,"threads_started":1}
I20260927 13:43:37.970703 8135 tablet_copy_service.cc:140] P 578daa7b1b23425589168c5d6c66bf0e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 041a10c1b226426c96f64572df2f027f ({username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:52434)
I20260927 13:43:37.970960 8135 tablet_copy_service.cc:161] P 578daa7b1b23425589168c5d6c66bf0e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 041a10c1b226426c96f64572df2f027f at {username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:52434: session id = 041a10c1b226426c96f64572df2f027f-00000000000000000000000000000000
I20260927 13:43:37.973729 8135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:43:37.975443 8576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:43:37.979851 8576 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: tablet copy: Starting download of 0 data blocks...
I20260927 13:43:37.980155 8576 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: tablet copy: Starting download of 1 WAL segments...
I20260927 13:43:37.982419 8576 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:43:37.985632 8135 tablet_copy_service.cc:342] P 578daa7b1b23425589168c5d6c66bf0e: Request end of tablet copy session 041a10c1b226426c96f64572df2f027f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:52434
I20260927 13:43:37.985807 8135 tablet_copy_service.cc:433] P 578daa7b1b23425589168c5d6c66bf0e: ending tablet copy session 041a10c1b226426c96f64572df2f027f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 041a10c1b226426c96f64572df2f027f
I20260927 13:43:37.995050 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:37.997428 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:38.003710 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:38.008633 8576 server_base.cc:1034] running on GCE node
W20260927 13:43:38.008904 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:38.009328 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:38.010282 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:38.011538 8576 hybrid_clock.cc:648] HybridClock initialized: now 1790516618011473 us; error 40 us; skew 500 ppm
I20260927 13:43:38.011765 8576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:38.013602 8576 init.cc:377] Logged in from keytab as oryx/127.1.237.253@KRBTEST.COM (short username oryx)
I20260927 13:43:38.014851 8576 webserver.cc:460] Webserver started at http://127.1.237.253:45637/ using document root <none> and password file <none>
I20260927 13:43:38.015192 8576 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:38.015308 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:38.017472 8576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 13:43:38.019050 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:38.019448 8576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 13:43:38.019609 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "041a10c1b226426c96f64572df2f027f"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "620a233e4504ca823ba586e085d3591f"
server_key_iv: "377dbe03d89f11563f14732fb431e46a"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.019778 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:38.030237 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:38.033432 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:38.069242 8576 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.253:38091
I20260927 13:43:38.069262 8747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.253:38091 every 8 connection(s)
I20260927 13:43:38.070132 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
I20260927 13:43:38.076838 8748 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:43:38.081923 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Bootstrap starting.
I20260927 13:43:38.094528 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:43:38.094946 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Bootstrap complete.
I20260927 13:43:38.095814 8748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } }
I20260927 13:43:38.096040 8748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 041a10c1b226426c96f64572df2f027f, State: Initialized, Role: LEARNER
I20260927 13:43:38.096257 8748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } }
I20260927 13:43:38.097010 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } }
I20260927 13:43:38.097311 8748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:38.097464 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: LEARNER
I20260927 13:43:38.113557 8762 catalog_manager.cc:1259] Loaded cluster ID: 5a274b1f63aa4803a437882c40afd6df
I20260927 13:43:38.113656 8762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: loading cluster ID for follower catalog manager: success
I20260927 13:43:38.116950 8762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: acquiring CA information for follower catalog manager: success
I20260927 13:43:38.119130 8762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:43:38.305518 8677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:38.305751 8677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:38.305895 8677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:38.313522 8677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:38.313612 8677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:38.313697 8677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:38.313792 8677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:38.313871 8677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:38.323709 8677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.252.keytab
--principal=oryx/127.1.237.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:41075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.237.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:38.323971 8571 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
I20260927 13:43:38.324983 8677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:38.326243 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:38.337011 8767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:38.337044 8768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:38.337327 8770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:38.337869 8677 server_base.cc:1034] running on GCE node
I20260927 13:43:38.338521 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:38.339845 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:38.341064 8677 hybrid_clock.cc:648] HybridClock initialized: now 1790516618341034 us; error 46 us; skew 500 ppm
I20260927 13:43:38.341382 8677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:38.344755 8677 init.cc:377] Logged in from keytab as oryx/127.1.237.252@KRBTEST.COM (short username oryx)
I20260927 13:43:38.346102 8677 webserver.cc:460] Webserver started at http://127.1.237.252:37031/ using document root <none> and password file <none>
I20260927 13:43:38.346729 8677 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:38.346863 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:38.347237 8677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:38.349906 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/instance:
uuid: "5811d4ee81764b10b5b794bbaefdf35e"
format_stamp: "Formatted at 2026-09-27 13:43:38 on dist-test-slave-7g3l"
server_key: "48886a614f4a464fc2e3e4b43c1228ca"
server_key_iv: "a755c326667856d37267d66b86c2cc99"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.350622 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal/instance:
uuid: "5811d4ee81764b10b5b794bbaefdf35e"
format_stamp: "Formatted at 2026-09-27 13:43:38 on dist-test-slave-7g3l"
server_key: "48886a614f4a464fc2e3e4b43c1228ca"
server_key_iv: "a755c326667856d37267d66b86c2cc99"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.355281 8677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260927 13:43:38.358400 8777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:38.359601 8677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:38.359751 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "5811d4ee81764b10b5b794bbaefdf35e"
format_stamp: "Formatted at 2026-09-27 13:43:38 on dist-test-slave-7g3l"
server_key: "48886a614f4a464fc2e3e4b43c1228ca"
server_key_iv: "a755c326667856d37267d66b86c2cc99"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.359920 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:38.374580 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:38.378670 8677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:38.378923 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:38.396941 8677 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.252:41075
I20260927 13:43:38.396970 8828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.252:41075 every 8 connection(s)
I20260927 13:43:38.398223 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
I20260927 13:43:38.401266 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8677
I20260927 13:43:38.401669 8829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:38.410066 8829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.252@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:38.423918 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.403122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.1.237.252:41075)
0927 13:43:38.403828 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.403839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.403873 (+ 34us) server_negotiation.cc:407] Connection header received
0927 13:43:38.405802 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.405820 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.406155 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.406879 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.408211 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.409373 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.410743 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.411619 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.411782 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.416061 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.416070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.416082 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.416119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.419981 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.420744 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.420749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.420754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.420948 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.421473 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.421476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.421478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.421896 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.422172 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.422603 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.422995 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
I20260927 13:43:38.426638 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.412547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.1.237.254:34881)
0927 13:43:38.412706 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.412713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.413319 (+ 606us) server_negotiation.cc:407] Connection header received
0927 13:43:38.413507 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.413513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.413580 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.413686 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.414585 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.415344 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.417067 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.417446 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.417613 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.421584 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.421589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.421592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.421619 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.424192 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.424794 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.424799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.424802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.424940 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.425320 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.425323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.425325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.425571 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.425667 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.426126 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.426375 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:43:38.426648 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.412498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52444)
0927 13:43:38.413198 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.413234 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.413388 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.413736 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.413742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.414245 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.414446 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.414453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.415470 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.415475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.416917 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.416932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.417659 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.417667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.417859 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.418374 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.418403 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.421402 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.424304 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.424309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.424313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.424678 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.425056 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.425062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.425065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.425220 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.425687 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.425693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.425855 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.426050 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.426322 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":421,"threads_started":1}
I20260927 13:43:38.427912 8829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.252@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:38.439481 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.428505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59216)
0927 13:43:38.428667 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.428685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.428832 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.429247 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.429250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.429511 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.429614 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.429621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.430807 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.430811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.431738 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.431747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.432372 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.432376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.432484 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.432917 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.432932 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.434647 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.437719 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.437722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.437725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.438042 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.438330 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.438333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.438335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.438443 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.438839 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.438842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.438936 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.439037 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.439275 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:43:38.439742 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.428541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.1.237.253:38091)
0927 13:43:38.429011 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.429015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.429031 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:43:38.429075 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.429078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.429128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.429219 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.429725 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.430696 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.431849 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.432256 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.432346 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.434802 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.434806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.434809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.434835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.437618 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.438132 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.438136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.438138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.438246 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.438530 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.438532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.438534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.438754 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.438861 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.439113 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.439318 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260927 13:43:38.440585 8829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.252@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
I20260927 13:43:38.451484 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.441100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.0.0.1:49686)
0927 13:43:38.441196 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.441208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.441273 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.441557 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.441560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.441831 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.441948 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.441954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.442792 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.442796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.443790 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.443800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.444473 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.444477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.444685 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.445164 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.445179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.446977 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.449665 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.449668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.449671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.449978 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.450267 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.450270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.450272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.450396 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.450795 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.450797 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.450899 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.451006 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.451207 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 13:43:38.451650 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.441226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.1.237.252:41075)
0927 13:43:38.441355 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.441358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.441373 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:43:38.441406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.441409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.441453 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.441523 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.442089 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.442697 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.443922 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.444347 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.444447 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.447129 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.447133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.447136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.447406 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.449548 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.450080 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.450084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.450086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.450179 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.450480 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.450483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.450485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.450707 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.450803 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.451078 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.451326 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":47}
I20260927 13:43:38.455705 8829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Bootstrap starting.
I20260927 13:43:38.458627 8829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:38.460017 8829 log.cc:824] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:38.462320 8829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: No bootstrap required, opened a new log
I20260927 13:43:38.468730 8829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } }
I20260927 13:43:38.469002 8829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:38.469076 8829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5811d4ee81764b10b5b794bbaefdf35e, State: Initialized, Role: FOLLOWER
I20260927 13:43:38.469493 8829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } }
I20260927 13:43:38.470201 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } } }
I20260927 13:43:38.470515 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:38.471056 8829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:38.478245 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8176
W20260927 13:43:38.483453 8847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:43:38.483589 8847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:43:38.484524 8677 master_runner.cc:428] Detected that this master 5811d4ee81764b10b5b794bbaefdf35e is joining an existing cluster
I20260927 13:43:38.484628 8677 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.237.252:41075
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.193@KRBTEST.COM
I20260927 13:43:38.500272 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.481668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.1.237.253:38091)
0927 13:43:38.481826 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.481833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.493180 (+ 11347us) server_negotiation.cc:407] Connection header received
0927 13:43:38.493574 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.493579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.493690 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.493767 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:38.494350 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.495411 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.497704 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.498994 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.499092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.499310 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.499864 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.500016 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:43:38.500967 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.481267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59224)
0927 13:43:38.493086 (+ 11819us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.493107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.493408 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.493970 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.493975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.494000 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:38.494205 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.494218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.495587 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.495592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.497528 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.497542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.499470 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.499477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.499659 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.499717 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.500526 (+ 809us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.500659 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11711,"thread_start_us":11925,"threads_started":1}
I20260927 13:43:38.501932 8723 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 13:43:38.502616 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 578daa7b1b23425589168c5d6c66bf0e. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } }
I20260927 13:43:38.502844 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: LEARNER
I20260927 13:43:38.505539 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.479271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.1.237.193:33031)
0927 13:43:38.480371 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.480379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.480406 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:43:38.480473 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.480482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.481111 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.482923 (+ 1812us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.483194 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.484653 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.486863 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.488009 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.488913 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.495531 (+ 6618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.495548 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.495568 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.495623 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.501403 (+ 5780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.502518 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.502529 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.502536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.502703 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.503414 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.503422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.503428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.503919 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.504284 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.504781 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.505058 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":4048,"threads_started":1}
I20260927 13:43:38.507059 8299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:38.527017 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.508175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.1.237.254:34881)
0927 13:43:38.508379 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.508388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.508458 (+ 70us) server_negotiation.cc:407] Connection header received
0927 13:43:38.508696 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.508701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.508774 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.508835 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.510106 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.511563 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.514595 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.515476 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.515602 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.518975 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.518984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.518990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.519035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.523112 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.524122 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.524130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.524136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.524347 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.524893 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.524901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.524906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.525291 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.525429 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.526211 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.526502 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:43:38.528821 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.507729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52452)
0927 13:43:38.508356 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.508374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.508554 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.509017 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.509021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.509612 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.509944 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.509956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.512129 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.512137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.514386 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.514405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.515922 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.515931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.516614 (+ 683us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.517307 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.517338 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.518727 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.523342 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.523350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.523359 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.523951 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.524527 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.524535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.524540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.524748 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.525553 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.525560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.525852 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.527316 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.527809 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":651,"threads_started":1}
I20260927 13:43:38.543684 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.511831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.1.237.253:38091)
0927 13:43:38.512040 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.512048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.519317 (+ 7269us) server_negotiation.cc:407] Connection header received
0927 13:43:38.519534 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.519541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.519612 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.519677 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.527578 (+ 7901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.529055 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.531232 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.531861 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.532003 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.535589 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.535595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.535599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.535637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.539511 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.540262 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.540267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.540271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.540421 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.541655 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.541663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.541668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.541984 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.542122 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.543095 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.543343 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:43:38.544750 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.511675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59234)
0927 13:43:38.519231 (+ 7556us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.519250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.519398 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.519972 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.519979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.520767 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.527390 (+ 6623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.527402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.529277 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.529283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.530947 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.530959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.532277 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.532282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.532504 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.533691 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.533715 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.535389 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.539675 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.539680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.539684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.540126 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.540551 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.540556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.540559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.541507 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.542183 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.542188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.542406 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.542631 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.544084 (+ 1453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7403,"mutex_wait_us":21,"spinlock_wait_cycles":25344,"thread_start_us":7280,"threads_started":1}
I20260927 13:43:38.550889 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.519391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.1.237.252:41075)
0927 13:43:38.519577 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.519583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.526341 (+ 6758us) server_negotiation.cc:407] Connection header received
0927 13:43:38.526662 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.526667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.526723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.526779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.528739 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.529823 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.532037 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.532819 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.532917 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.544260 (+ 11343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.544267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.544272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.544306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.548161 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.548931 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.548936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.548940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.549089 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.549444 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.549448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.549452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.549761 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.549896 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.550356 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.550554 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":57}
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.254@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
I20260927 13:43:38.552034 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.519679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.0.0.1:49690)
0927 13:43:38.526254 (+ 6575us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.526271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.526545 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.526933 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.526937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.527874 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.528589 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.528600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.530370 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.530375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.531890 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.531919 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.535571 (+ 3652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.535578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.535759 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.540936 (+ 5177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.540961 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.542536 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.548321 (+ 5785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.548327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.548336 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.548809 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.549210 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.549214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.549218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.549336 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.549959 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.549964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.550110 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.551335 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.551600 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6459,"thread_start_us":6702,"threads_started":1}
I20260927 13:43:38.559120 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.536943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.0.0.1:49704)
0927 13:43:38.537675 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.537692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.537786 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.542688 (+ 4902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.542693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.543117 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.543311 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.543323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.545696 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.545702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.547095 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.547110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.548332 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.548339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.548515 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.549157 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.549182 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.551917 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.556702 (+ 4785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.556707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.556711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.557160 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.557548 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.557553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.557556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.557748 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.558300 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.558304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.558470 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.558598 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.558824 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616}
I20260927 13:43:38.559289 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.537291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.1.237.252:41075)
0927 13:43:38.541534 (+ 4243us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.541540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.541560 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:43:38.541638 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.541643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.541714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.543475 (+ 1761us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.544194 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.545554 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.547316 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.548128 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.552114 (+ 3986us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.552357 (+ 243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.552365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.552370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.552402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.556552 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.557299 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.557304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.557307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.557433 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.557851 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.557856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.557859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.558154 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.558304 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.558722 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.558883 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4124,"thread_start_us":3269,"threads_started":1}
I20260927 13:43:38.560420 8114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.237.252:41075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e"
member_type: NON_VOTER
last_known_addr {
host: "127.1.237.252"
port: 41075
}
attrs {
promote: true
}
}
dest_uuid: "578daa7b1b23425589168c5d6c66bf0e"
cas_config_opid_index: 5
I20260927 13:43:38.561403 8114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } }
I20260927 13:43:38.563012 8849 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.566761 8849 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.237.252:41075
I20260927 13:43:38.567750 8723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 13:43:38.567983 8804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:38.571234 8804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 13:43:38.571134 8849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 13:43:38.571843 8677 master.cc:537] Master@127.1.237.252:41075 shutting down...
I20260927 13:43:38.571895 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.572410 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:38.581205 8849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 13:43:38.584335 8723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.588740 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: NON_VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: true } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.589025 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: LEARNER
I20260927 13:43:38.589255 8849 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: attempting to promote NON_VOTER 041a10c1b226426c96f64572df2f027f to VOTER
I20260927 13:43:38.589919 8849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } }
I20260927 13:43:38.591674 8723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 LEARNER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 13:43:38.592171 8862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 13:43:38.594319 8861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 041a10c1b226426c96f64572df2f027f (127.1.237.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.594913 8723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 041a10c1b226426c96f64572df2f027f (127.1.237.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.598652 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.598917 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:38.598922 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.599176 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:38.599300 8849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.599498 8849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:38.612555 8677 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:38.612973 8677 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:38.613176 8677 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: stopping tablet replica
W20260927 13:43:38.615995 8100 consensus_peers.cc:487] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e -> Peer 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252:41075): Couldn't send request to peer 5811d4ee81764b10b5b794bbaefdf35e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 13:43:38.629984 8677 master.cc:552] Master@127.1.237.252:41075 shutdown complete.
I20260927 13:43:38.630074 8677 master_runner.cc:305] Clearing existing system tablet
I20260927 13:43:38.630452 8677 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:38.630540 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:38.633719 8677 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:38.635321 8867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:38.635633 8677 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:38.635893 8677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 13:43:38.636000 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "5811d4ee81764b10b5b794bbaefdf35e"
format_stamp: "Formatted at 2026-09-27 13:43:38 on dist-test-slave-7g3l"
server_key: "48886a614f4a464fc2e3e4b43c1228ca"
server_key_iv: "a755c326667856d37267d66b86c2cc99"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.640808 8677 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:38.647588 8677 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 13:43:38.647719 8677 log.cc:1196] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 13:43:38.648141 8677 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Deleting consensus metadata
I20260927 13:43:38.648402 8677 master_runner.cc:315] Copying system tablet from 127.1.237.254:34881
I20260927 13:43:38.657058 8677 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: tablet copy: Beginning tablet copy session from remote peer at address 127.1.237.254:34881
I20260927 13:43:38.666390 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8176
I20260927 13:43:38.674479 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.657798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.1.237.254:34881)
0927 13:43:38.658074 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.658081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.659734 (+ 1653us) server_negotiation.cc:407] Connection header received
0927 13:43:38.659881 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.659888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.659991 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.660102 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:38.661086 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.662235 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.664260 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.664869 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.665004 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.667959 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:38.667965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:38.667971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:38.668008 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:38.671410 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.672239 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.672245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.672249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.672403 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:38.672841 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:38.672846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:38.672850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:38.673171 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:38.673304 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.673842 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.674104 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 13:43:38.675498 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.657788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52462)
0927 13:43:38.659610 (+ 1822us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.659634 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.659773 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.660148 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.660153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.660655 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:38.660915 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.660927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.662382 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.662387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.664068 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.664086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.665767 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.665776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.665925 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.666635 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:38.666659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:38.667727 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:38.671557 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.671562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.671566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.672111 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.672541 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:38.672546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:38.672550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:38.672712 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:38.673388 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:38.673393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:38.673569 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.674748 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.675052 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"thread_start_us":1574,"threads_started":1}
I20260927 13:43:38.676164 8135 tablet_copy_service.cc:140] P 578daa7b1b23425589168c5d6c66bf0e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5811d4ee81764b10b5b794bbaefdf35e ({username='oryx', principal='oryx/127.1.237.252@KRBTEST.COM'} at 127.0.0.1:52462)
I20260927 13:43:38.676316 8135 tablet_copy_service.cc:161] P 578daa7b1b23425589168c5d6c66bf0e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5811d4ee81764b10b5b794bbaefdf35e at {username='oryx', principal='oryx/127.1.237.252@KRBTEST.COM'} at 127.0.0.1:52462: session id = 5811d4ee81764b10b5b794bbaefdf35e-00000000000000000000000000000000
I20260927 13:43:38.679919 8135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 13:43:38.681692 8677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 13:43:38.685480 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:33031
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:34881
--webserver_port=39927
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:43:38.688969 8677 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: tablet copy: Starting download of 0 data blocks...
I20260927 13:43:38.689229 8677 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: tablet copy: Starting download of 1 WAL segments...
I20260927 13:43:38.691458 8677 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 13:43:38.695497 8135 tablet_copy_service.cc:342] P 578daa7b1b23425589168c5d6c66bf0e: Request end of tablet copy session 5811d4ee81764b10b5b794bbaefdf35e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.237.252@KRBTEST.COM'} at 127.0.0.1:52462
I20260927 13:43:38.695626 8135 tablet_copy_service.cc:433] P 578daa7b1b23425589168c5d6c66bf0e: ending tablet copy session 5811d4ee81764b10b5b794bbaefdf35e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5811d4ee81764b10b5b794bbaefdf35e
I20260927 13:43:38.707139 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:38.709587 8875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:38.710445 8876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:38.717465 8878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:38.718080 8677 server_base.cc:1034] running on GCE node
I20260927 13:43:38.718406 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:38.719007 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:38.720197 8677 hybrid_clock.cc:648] HybridClock initialized: now 1790516618720166 us; error 22 us; skew 500 ppm
I20260927 13:43:38.720415 8677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:38 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516618, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:38.722508 8677 init.cc:377] Logged in from keytab as oryx/127.1.237.252@KRBTEST.COM (short username oryx)
I20260927 13:43:38.727975 8677 webserver.cc:460] Webserver started at http://127.1.237.252:45961/ using document root <none> and password file <none>
I20260927 13:43:38.728317 8677 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:38.728466 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:38.731314 8677 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:43:38.733157 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:38.733623 8677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 13:43:38.733775 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "5811d4ee81764b10b5b794bbaefdf35e"
format_stamp: "Formatted at 2026-09-27 13:43:38 on dist-test-slave-7g3l"
server_key: "48886a614f4a464fc2e3e4b43c1228ca"
server_key_iv: "a755c326667856d37267d66b86c2cc99"
server_key_version: "encryptionkey@0"
I20260927 13:43:38.734015 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:38.754763 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:38.758514 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:38.779028 8677 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.252:41075
I20260927 13:43:38.779078 8936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.252:41075 every 8 connection(s)
I20260927 13:43:38.780082 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
I20260927 13:43:38.785045 8937 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:43:38.788744 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Bootstrap starting.
I20260927 13:43:38.806478 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:43:38.807022 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: Bootstrap complete.
I20260927 13:43:38.808144 8937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } }
I20260927 13:43:38.808451 8937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5811d4ee81764b10b5b794bbaefdf35e, State: Initialized, Role: LEARNER
I20260927 13:43:38.808744 8937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } }
I20260927 13:43:38.809523 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.809832 8937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:38.810014 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: LEARNER
I20260927 13:43:38.818354 8951 catalog_manager.cc:1259] Loaded cluster ID: 5a274b1f63aa4803a437882c40afd6df
I20260927 13:43:38.818511 8951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: loading cluster ID for follower catalog manager: success
I20260927 13:43:38.822006 8951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: acquiring CA information for follower catalog manager: success
I20260927 13:43:38.824307 8951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:43:38.987782 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.978990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.0.0.1:49712)
0927 13:43:38.979204 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:38.979225 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:38.979403 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:38.980163 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:38.980170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:38.980202 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:38.980402 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.980414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.982720 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.982727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.984947 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:38.984966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.986873 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.986880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:38.987115 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.987189 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 13:43:38.987299 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 13:43:38.987464 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:43:38.988185 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:38.979199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.1.237.252:41075)
0927 13:43:38.979790 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:38.979794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:38.979813 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:43:38.979876 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:38.979902 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:38.979994 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:38.980102 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:38.981026 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.982537 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.985176 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:38.986640 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:38.986841 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:38.987130 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:38.987443 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 13:43:38.987625 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":327,"threads_started":1}
I20260927 13:43:38.989084 8912 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 13:43:38.989745 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 578daa7b1b23425589168c5d6c66bf0e. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:38.990026 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: LEARNER
W20260927 13:43:39.040531 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:39.040820 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:39.040985 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:39.048403 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:39.048532 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:39.048703 8874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:39.058638 8874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:33031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=39927
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:39.059861 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:39.061075 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:39.077926 8957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:39.077931 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:39.084776 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:39.085575 8874 server_base.cc:1034] running on GCE node
I20260927 13:43:39.086172 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:39.087109 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:39.088337 8874 hybrid_clock.cc:648] HybridClock initialized: now 1790516619088281 us; error 47 us; skew 500 ppm
I20260927 13:43:39.088639 8874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:39.091609 8874 init.cc:377] Logged in from keytab as oryx/127.1.237.193@KRBTEST.COM (short username oryx)
I20260927 13:43:39.094609 8874 webserver.cc:460] Webserver started at http://127.1.237.193:39927/ using document root <none> and password file <none>
I20260927 13:43:39.095762 8874 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:39.095908 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:39.101341 8874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 13:43:39.104041 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.105410 8874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:43:39.105583 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a1526686335d49e7ba21bd9cb437eb1f"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "47eea133a4cd1672a3dc9e7eb8d81ceb"
server_key_iv: "4f246399710d7841095d60aeebc7361f"
server_key_version: "encryptionkey@0"
I20260927 13:43:39.106737 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:39.125231 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:39.129633 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:39.129890 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:39.130774 8874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:39.132153 8874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:39.132267 8874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.132391 8874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:39.132473 8874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.174073 8874 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:33031
I20260927 13:43:39.174106 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:33031 every 8 connection(s)
I20260927 13:43:39.175508 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:39.177450 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 8874
I20260927 13:43:39.177810 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8310
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.194@KRBTEST.COM
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:39.228261 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.179351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.1.237.194:43445)
0927 13:43:39.206781 (+ 27430us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.206786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.206806 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:43:39.206877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.206882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.207053 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.207270 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.209244 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.210801 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.212598 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.213421 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.213538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.220031 (+ 6493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.220051 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.220067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.220105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.224395 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.225203 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.225214 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.225220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.225364 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.225775 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.225780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.225784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.226214 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.226485 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.227521 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.227790 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27303,"thread_start_us":28099,"threads_started":1}
I20260927 13:43:39.229562 8433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:39.229774 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.179735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52473 (local address 127.1.237.254:34881)
0927 13:43:39.206344 (+ 26609us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.206349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.206365 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:43:39.207785 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.207791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.207874 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.207987 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.210431 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.211623 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.214562 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.215220 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.215325 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.220144 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.220149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.220155 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.220192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.224385 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.225298 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.225304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.225307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.225435 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.226160 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.226165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.226169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.226492 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.226609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.227927 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.228137 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26513,"thread_start_us":26620,"threads_started":1}
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
I20260927 13:43:39.233250 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.180420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:52473)
0927 13:43:39.192511 (+ 12091us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.192555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.207581 (+ 15026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.208455 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.208469 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.209278 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.210157 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.210185 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.211797 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.211802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.214373 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.214394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.215641 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.215649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.215959 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.216709 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.216743 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.219845 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.224621 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.224631 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.224658 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.225177 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.225583 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.225588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.225591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.226034 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.226712 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.226720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.227113 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.230455 (+ 3342us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.231224 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11728,"thread_start_us":11760,"threads_started":1}
I20260927 13:43:39.235478 9081 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:39.235801 9081 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:39.237126 9081 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
I20260927 13:43:39.240116 8114 ts_manager.cc:194] Re-registered known tserver with Master: a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031)
I20260927 13:43:39.241415 8114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:52473
I20260927 13:43:39.247504 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.185589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.1.237.193:55849)
0927 13:43:39.198767 (+ 13178us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.198787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.203508 (+ 4721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.210997 (+ 7489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.211004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.211567 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.212452 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.212466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.217873 (+ 5407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.217879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.219782 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.219798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.220929 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.220937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.221157 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.221680 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.221700 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.224699 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.244646 (+ 19947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.244653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.244659 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.245222 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.245755 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.245762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.245769 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.245943 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.246516 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.246520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.246649 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.246775 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.247049 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12827,"thread_start_us":11770,"threads_started":1}
I20260927 13:43:39.248189 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.198014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:33187 (local address 127.1.237.252:41075)
0927 13:43:39.198235 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.198243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.209096 (+ 10853us) server_negotiation.cc:407] Connection header received
0927 13:43:39.209327 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.209334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.209416 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.209487 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.214287 (+ 4800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.215555 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.225893 (+ 10338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.226696 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.226810 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.233790 (+ 6980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.233798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.233805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.233850 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.243547 (+ 9697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.245973 (+ 2426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.245978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.245984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.246157 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.246566 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.246571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.246575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.246933 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.247120 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.247644 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.247869 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:43:39.248911 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.185600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:55849 (local address 127.1.237.253:38091)
0927 13:43:39.208235 (+ 22635us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.208244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.208273 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:43:39.208346 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.208353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.208455 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.212580 (+ 4125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.212805 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.214088 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.219987 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.220735 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.221455 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.224999 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.225007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.225013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.225048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.232651 (+ 7603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.245443 (+ 12792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.245451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.245456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.245602 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.246101 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.246106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.246109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.246383 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.246537 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.246875 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.247353 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22476,"thread_start_us":23275,"threads_started":1}
I20260927 13:43:39.249058 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.198374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.1.237.193:33187)
0927 13:43:39.208998 (+ 10624us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.209016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.209192 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.209703 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.209710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.210390 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.210782 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.210798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.215752 (+ 4954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.215759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.217690 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.217707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.227134 (+ 9427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.227142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.227374 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.228685 (+ 1311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.228710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.233550 (+ 4840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.243746 (+ 10196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.243753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.243758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.244423 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.246306 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.246310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.246314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.246438 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.247229 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.247233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.247367 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.248350 (+ 983us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.248678 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10494,"mutex_wait_us":108,"thread_start_us":10228,"threads_started":1}
I20260927 13:43:39.249768 9082 heartbeater.cc:347] Connected to a master server at 127.1.237.253:38091
I20260927 13:43:39.249893 9082 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:39.250298 9080 heartbeater.cc:347] Connected to a master server at 127.1.237.252:41075
I20260927 13:43:39.250326 9082 heartbeater.cc:511] Master 127.1.237.253:38091 requested a full tablet report, sending...
I20260927 13:43:39.250422 9080 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:39.250751 9080 heartbeater.cc:511] Master 127.1.237.252:41075 requested a full tablet report, sending...
I20260927 13:43:39.252065 8713 ts_manager.cc:194] Registered new tserver with Master: a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031)
I20260927 13:43:39.252391 8902 ts_manager.cc:194] Registered new tserver with Master: a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031)
I20260927 13:43:39.326115 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8310
I20260927 13:43:39.342831 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:43445
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:34881
--webserver_port=37345
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:43:39.379735 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: NON_VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: true } } }
I20260927 13:43:39.380070 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:39.381284 8861 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: attempting to promote NON_VOTER 5811d4ee81764b10b5b794bbaefdf35e to VOTER
I20260927 13:43:39.381978 8861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } }
I20260927 13:43:39.383876 8912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 LEARNER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:43:39.383926 8723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Refusing update from remote peer 578daa7b1b23425589168c5d6c66bf0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 13:43:39.384434 8861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:43:39.384773 9093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 13:43:39.387259 8861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.387818 8723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.392292 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.392652 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:39.392879 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.393146 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:39.395901 8911 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.400030 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.400270 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:39.401062 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.400688 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "578daa7b1b23425589168c5d6c66bf0e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:39.401345 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:39.401435 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 13:43:39.659307 9092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:39.659562 9092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:39.659715 9092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:39.666443 9092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:39.666528 9092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:39.666636 9092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:39.675663 9092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:43445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=37345
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:39.676750 9092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:39.677886 9092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:39.689846 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:39.690055 9105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:39.690501 9102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:39.690466 9092 server_base.cc:1034] running on GCE node
I20260927 13:43:39.691128 9092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:39.692075 9092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:39.693262 9092 hybrid_clock.cc:648] HybridClock initialized: now 1790516619693210 us; error 54 us; skew 500 ppm
I20260927 13:43:39.693552 9092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:39.696591 9092 init.cc:377] Logged in from keytab as oryx/127.1.237.194@KRBTEST.COM (short username oryx)
I20260927 13:43:39.697863 9092 webserver.cc:460] Webserver started at http://127.1.237.194:37345/ using document root <none> and password file <none>
I20260927 13:43:39.698523 9092 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:39.698648 9092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:39.703464 9092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:43:39.705924 9112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.707111 9092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:39.707288 9092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "867bbe567f7f431b88e06f660bfc65a7"
format_stamp: "Formatted at 2026-09-27 13:43:36 on dist-test-slave-7g3l"
server_key: "29706c833a7fdeaa8804efc2616e4e5a"
server_key_iv: "4fe2224a72de100337a84de73b062967"
server_key_version: "encryptionkey@0"
I20260927 13:43:39.708114 9092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:39.726730 9092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:39.730355 9092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:39.730650 9092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:39.731592 9092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:39.732738 9092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:39.732823 9092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.732896 9092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:39.732949 9092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:39.763638 9092 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:43445
I20260927 13:43:39.763722 9224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:43445 every 8 connection(s)
I20260927 13:43:39.764874 9092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:39.771740 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9092
I20260927 13:43:39.772042 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8443
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
I20260927 13:43:39.801151 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.767359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:39179 (local address 127.1.237.254:34881)
0927 13:43:39.776322 (+ 8963us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.776341 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.776364 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:43:39.776440 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.776448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.776554 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.776681 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.779170 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.780195 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.782427 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.783058 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.783856 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.789184 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.789192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.789197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.789237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.793492 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.796515 (+ 3023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.796521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.796524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.796667 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.797414 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.797419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.797423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.797751 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.797881 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.799794 (+ 1913us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.800778 (+ 984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8779,"thread_start_us":8664,"threads_started":1}
I20260927 13:43:39.802742 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.768170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:39179)
0927 13:43:39.775554 (+ 7384us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.775576 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.775697 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.777176 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.777191 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.778198 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.778940 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.778965 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.780367 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.780372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.782260 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.782273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.783289 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.783295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.783535 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.784684 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.784709 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.788952 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.793644 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.793652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.793665 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.794222 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.797027 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.797032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.797036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.797267 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.797901 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.797912 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.798794 (+ 882us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.799182 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.801426 (+ 2244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7181,"mutex_wait_us":387,"thread_start_us":2713,"threads_started":1}
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.195@KRBTEST.COM
I20260927 13:43:39.805178 9226 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:39.805550 9226 heartbeater.cc:464] Registering TS with master...
Sep 27 13:43:39 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516619, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:39.806442 9226 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
I20260927 13:43:39.808336 8114 ts_manager.cc:194] Re-registered known tserver with Master: 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:39.808563 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.768170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.1.237.194:57359)
0927 13:43:39.771762 (+ 3592us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.771809 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.774697 (+ 2888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.786552 (+ 11855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.786557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.786937 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.787184 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.787195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.794362 (+ 7167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.794367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.795615 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.795626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.796350 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.796356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.796542 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.798027 (+ 1485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.798047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.801271 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.805389 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.805393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.805396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.805904 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.806336 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.806341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.806344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.806956 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.807650 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.807654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.807821 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.807959 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.808198 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3290,"thread_start_us":6908,"threads_started":1}
I20260927 13:43:39.809371 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.767359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:57359 (local address 127.1.237.252:41075)
0927 13:43:39.786232 (+ 18873us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.786238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.786255 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:39.786302 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.786307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.786369 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.787323 (+ 954us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.787527 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.791806 (+ 4279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.795756 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.796180 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.796318 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.801490 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.801498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.801503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.801543 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.805263 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.806063 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.806068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.806072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.806202 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.807076 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.807081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.807085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.807434 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.807581 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.808818 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.809029 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18753,"thread_start_us":19422,"threads_started":1}
I20260927 13:43:39.809408 8114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:39179
I20260927 13:43:39.810636 9225 heartbeater.cc:347] Connected to a master server at 127.1.237.252:41075
I20260927 13:43:39.810752 9225 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:39.811233 9225 heartbeater.cc:511] Master 127.1.237.252:41075 requested a full tablet report, sending...
I20260927 13:43:39.812491 8902 ts_manager.cc:194] Registered new tserver with Master: 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:39.814663 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.773266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.1.237.195:39839)
0927 13:43:39.791748 (+ 18482us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.791760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.791792 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:43:39.791870 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.791880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.792165 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.794853 (+ 2688us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.795136 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.796786 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.798676 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.799578 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.800212 (+ 634us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.807740 (+ 7528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.807752 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.807768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.807801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.811601 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.812343 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.812351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.812354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.812506 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.812988 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.812997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.813002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.813378 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.813645 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.814082 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.814322 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18269,"thread_start_us":19030,"threads_started":1}
I20260927 13:43:39.815649 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.776941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.1.237.194:40891)
0927 13:43:39.796747 (+ 19806us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:39.796766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:39.796889 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:39.797536 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:39.797540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:39.798104 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:39.799043 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.799054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.800517 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.800522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.802048 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:39.802063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.803263 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.803269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:39.803855 (+ 586us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.804371 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:39.804401 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:39.807356 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:39.812510 (+ 5154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.812514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.812517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.813367 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.813800 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:39.813805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:39.813809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:39.814016 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:39.814664 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:39.814668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:39.814808 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 13:43:39.814941 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:43:39.815247 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19692,"mutex_wait_us":22,"thread_start_us":20017,"threads_started":1}
I20260927 13:43:39.815691 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:39.775882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:40891 (local address 127.1.237.253:38091)
0927 13:43:39.796335 (+ 20453us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:39.796341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:39.796840 (+ 499us) server_negotiation.cc:407] Connection header received
0927 13:43:39.797293 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:39.797299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:39.797400 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:39.797523 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:39.799223 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.800286 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.802203 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:39.802789 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:39.802900 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:39.807541 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:39.807547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:39.807553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:39.807606 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:39.812107 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.813498 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.813502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.813504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.813666 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:39.814148 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:39.814153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:39.814156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:39.814448 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:39.814600 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:39.815074 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 13:43:39.815323 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20327,"thread_start_us":21217,"threads_started":1}
I20260927 13:43:39.815814 8566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:39.816459 9227 heartbeater.cc:347] Connected to a master server at 127.1.237.253:38091
I20260927 13:43:39.816602 9227 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:39.817112 9227 heartbeater.cc:511] Master 127.1.237.253:38091 requested a full tablet report, sending...
I20260927 13:43:39.818346 8713 ts_manager.cc:194] Registered new tserver with Master: 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:39.895386 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8443
I20260927 13:43:39.911648 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:39839
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:34881
--webserver_port=42121
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:40.221222 9238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:40.221457 9238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:40.221633 9238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:40.228242 9238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:40.228358 9238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:40.228511 9238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:40.237336 9238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:39839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=42121
--tserver_master_addrs=127.1.237.254:34881,127.1.237.253:38091,127.1.237.252:41075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:40.238464 9238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:40.239669 9238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:43:40.245759 9081 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
W20260927 13:43:40.252000 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:40.252080 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:40.252030 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:40.252928 9238 server_base.cc:1034] running on GCE node
I20260927 13:43:40.253608 9238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:40.254575 9238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:40.255774 9238 hybrid_clock.cc:648] HybridClock initialized: now 1790516620255724 us; error 40 us; skew 500 ppm
I20260927 13:43:40.256055 9238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:40 dist-test-slave-7g3l krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:40.259076 9238 init.cc:377] Logged in from keytab as oryx/127.1.237.195@KRBTEST.COM (short username oryx)
I20260927 13:43:40.260321 9238 webserver.cc:460] Webserver started at http://127.1.237.195:42121/ using document root <none> and password file <none>
I20260927 13:43:40.260864 9238 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:40.260962 9238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:40.265734 9238 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 13:43:40.268035 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:40.269284 9238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 13:43:40.269398 9238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "3c4f17f14f644f47bd427309a83ad90b"
format_stamp: "Formatted at 2026-09-27 13:43:37 on dist-test-slave-7g3l"
server_key: "ea8a740e9d7202ac91b985d7581c3049"
server_key_iv: "c17111998139610becf0436618603f69"
server_key_version: "encryptionkey@0"
I20260927 13:43:40.270200 9238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:40.290175 9238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:40.294090 9238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:40.294351 9238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:40.295250 9238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:40.296494 9238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:40.296597 9238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:40.296725 9238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:40.296840 9238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:40.329063 9238 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:39839
I20260927 13:43:40.329155 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:39839 every 8 connection(s)
I20260927 13:43:40.330453 9238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:40.339361 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9238
Sep 27 13:43:40 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
Sep 27 13:43:40 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.252@KRBTEST.COM
Sep 27 13:43:40 dist-test-slave-7g3l krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.253@KRBTEST.COM
I20260927 13:43:40.366041 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.333691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:50587)
0927 13:43:40.334540 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.334587 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.338313 (+ 3726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.339857 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.339868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.340775 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:40.341750 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.341780 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.347317 (+ 5537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.347325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.350100 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.350119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.352321 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.352328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.352611 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.353727 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:40.353752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:40.356785 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:40.360929 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.360938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.360955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.361503 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.361928 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.361932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.361936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.362197 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.363203 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:40.363213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:40.363692 (+ 479us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.364000 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.364519 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":833,"threads_started":1}
I20260927 13:43:40.367219 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.333116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:50587 (local address 127.1.237.254:34881)
0927 13:43:40.333556 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.333562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.334725 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 13:43:40.338680 (+ 3955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.338689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.338795 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.338889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.345479 (+ 6590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.346972 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.351226 (+ 4254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.351836 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.351949 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.357057 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.357066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.357072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.357121 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.360784 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.361651 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.361658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.361662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.361801 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.362394 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.362399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.362404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.363036 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.366465 (+ 3429us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.366606 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.366813 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":220,"threads_started":1}
I20260927 13:43:40.368634 9368 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34881
I20260927 13:43:40.368949 9368 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:40.369560 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.341655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.1.237.195:39375)
0927 13:43:40.351241 (+ 9586us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.351258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.351355 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.351927 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.351932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.352600 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:40.352897 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.352911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.354289 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.354294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.355848 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.355861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.356947 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.356956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.357212 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.357837 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:40.357865 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:40.362740 (+ 4875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:40.366873 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.366878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.366882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.367410 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.367811 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.367815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.367818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.367944 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.368581 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:40.368585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:40.368750 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.368880 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.369143 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9452,"mutex_wait_us":348,"thread_start_us":8959,"threads_started":1}
I20260927 13:43:40.369859 9368 heartbeater.cc:511] Master 127.1.237.254:34881 requested a full tablet report, sending...
I20260927 13:43:40.370440 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.339809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:39375 (local address 127.1.237.252:41075)
0927 13:43:40.350765 (+ 10956us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.350774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.351517 (+ 743us) server_negotiation.cc:407] Connection header received
0927 13:43:40.351577 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.351582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.351657 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.351753 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.353083 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.354134 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.356022 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.356604 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.356719 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.362942 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.362948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.362952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.363033 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.366727 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.367549 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.367553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.367557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.367687 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.368066 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.368070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.368073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.368367 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.368524 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.369917 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.370123 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10807,"thread_start_us":11606,"threads_started":1}
I20260927 13:43:40.371193 9367 heartbeater.cc:347] Connected to a master server at 127.1.237.252:41075
I20260927 13:43:40.371308 9367 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:40.371668 9367 heartbeater.cc:511] Master 127.1.237.252:41075 requested a full tablet report, sending...
I20260927 13:43:40.371724 8114 ts_manager.cc:194] Re-registered known tserver with Master: 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:40.372715 8902 ts_manager.cc:194] Registered new tserver with Master: 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:40.372828 8114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:50587
I20260927 13:43:40.374820 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.334409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.1.237.195:46431)
0927 13:43:40.343146 (+ 8737us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.343187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.343312 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.356990 (+ 13678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.356995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.357574 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:40.357906 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.357918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.359518 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.359525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.361005 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.361024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.362023 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.362031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.362335 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.363028 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:40.363057 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:40.366654 (+ 3597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:40.371906 (+ 5252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.371911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.371915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.372360 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.372797 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:40.372804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:40.372808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:40.372955 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:40.373666 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:40.373670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:40.373860 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.374050 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.374392 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7578,"thread_start_us":4927,"threads_started":1}
I20260927 13:43:40.376106 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.334418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:46431 (local address 127.1.237.253:38091)
0927 13:43:40.355760 (+ 21342us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.355767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.355786 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:43:40.355838 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.355842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.355927 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.358073 (+ 2146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.358212 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.359336 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.361183 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.361804 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.361952 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.366907 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.366916 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.366923 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.366984 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.371702 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.372519 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.372523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.372526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.372669 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.373098 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.373103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.373107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.373454 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.373597 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.375340 (+ 1743us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.375607 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21223,"thread_start_us":21113,"threads_started":1}
I20260927 13:43:40.377245 9369 heartbeater.cc:347] Connected to a master server at 127.1.237.253:38091
I20260927 13:43:40.377385 9369 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:40.377791 9369 heartbeater.cc:511] Master 127.1.237.253:38091 requested a full tablet report, sending...
I20260927 13:43:40.379098 8713 ts_manager.cc:194] Registered new tserver with Master: 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:40.399195 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.373780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.1.237.254:34881)
0927 13:43:40.376536 (+ 2756us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.376544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.383317 (+ 6773us) server_negotiation.cc:407] Connection header received
0927 13:43:40.383770 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.383774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.383837 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.383898 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.386726 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.387854 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.389943 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.390549 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.390665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.393852 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.393858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.393862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.393902 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.396569 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.397181 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.397187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.397190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.397334 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.397784 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.397788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.397792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.398090 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.398291 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.398653 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.398838 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2629,"thread_start_us":5637,"threads_started":1}
I20260927 13:43:40.400568 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.375275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.1.237.253:38091)
0927 13:43:40.376630 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.376635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.384325 (+ 7690us) server_negotiation.cc:407] Connection header received
0927 13:43:40.384578 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.384583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.384671 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.384740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.387012 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.387771 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.389943 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.390549 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.390642 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.394650 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.394657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.394662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.394700 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.397812 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.398380 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.398384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.398388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.398504 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.398926 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.398931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.398935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.399244 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.399452 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.400045 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.400240 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":7045,"threads_started":1}
I20260927 13:43:40.400935 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.383391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.1.237.252:41075)
0927 13:43:40.383531 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.383537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.384803 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 13:43:40.384974 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.384979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.385075 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.385171 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.386849 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.387908 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.389321 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.389713 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.389831 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.393901 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.393909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.393914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.393951 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.397073 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.398356 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.398362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.398365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.398499 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.398926 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.398931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.398935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.399265 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.399459 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.400047 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.400238 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:43:40.403312 8114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:40.409893 8114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:40.443984 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.427829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33031 (local address 127.0.0.1:46624)
0927 13:43:40.436219 (+ 8390us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.436240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.436372 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.437395 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.437402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.437439 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:40.437860 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.437875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.439502 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.439507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.441590 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.441601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.443242 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.443249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.443408 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.443441 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.443559 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.443676 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8248,"thread_start_us":13668,"threads_started":1}
I20260927 13:43:40.444046 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.428137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.1.237.193:33031)
0927 13:43:40.434717 (+ 6580us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.434730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.436357 (+ 1627us) server_negotiation.cc:407] Connection header received
0927 13:43:40.436538 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.436550 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.436870 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.437128 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:40.438092 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.439362 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.441728 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.443026 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.443188 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.443390 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.443606 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.443745 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6318,"thread_start_us":6183,"threads_started":1}
I20260927 13:43:40.448114 9015 tablet_service.cc:1467] Processing CreateTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (DEFAULT_TABLE table=test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:40.449561 9015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:40.450259 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.427519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:43445 (local address 127.0.0.1:33320)
0927 13:43:40.435417 (+ 7898us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.435442 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.435591 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.442683 (+ 7092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.442689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.442715 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:40.443132 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.443144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.445132 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.445140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.447144 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.447195 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.449549 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.449555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.449706 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.449743 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.449820 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.449932 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7555,"mutex_wait_us":3,"thread_start_us":7383,"threads_started":1}
I20260927 13:43:40.450289 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.427677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.1.237.194:43445)
0927 13:43:40.442104 (+ 14427us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.442112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.442134 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:43:40.442218 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.442225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.442465 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.443327 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:40.443464 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.445009 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.447310 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.449367 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.449524 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.449708 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.449886 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.450042 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14255,"thread_start_us":14321,"threads_started":1}
I20260927 13:43:40.451571 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.435355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39839 (local address 127.0.0.1:37472)
0927 13:43:40.441783 (+ 6428us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.441802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.441919 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.442501 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.442508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.442534 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:40.442996 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.443013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.446550 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.446554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.449008 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.449021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.450780 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.450786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.450935 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.450964 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.451071 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.451212 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6321,"mutex_wait_us":19,"thread_start_us":556,"threads_started":1}
I20260927 13:43:40.451907 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.435717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.1.237.195:39839)
0927 13:43:40.438905 (+ 3188us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.438916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.441900 (+ 2984us) server_negotiation.cc:407] Connection header received
0927 13:43:40.442055 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.442062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.442293 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.442492 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:40.444501 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.445774 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.449248 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.450595 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.450726 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.450902 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.451422 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.451573 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2987,"thread_start_us":2776,"threads_started":1}
I20260927 13:43:40.453529 9160 tablet_service.cc:1467] Processing CreateTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (DEFAULT_TABLE table=test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:40.454741 9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:40.454588 9302 tablet_service.cc:1467] Processing CreateTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (DEFAULT_TABLE table=test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:40.455744 9302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:40.465953 9403 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Bootstrap starting.
I20260927 13:43:40.468382 9404 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Bootstrap starting.
I20260927 13:43:40.470055 9403 tablet_bootstrap.cc:654] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:40.470330 9405 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Bootstrap starting.
I20260927 13:43:40.471290 9403 log.cc:824] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:40.472188 9404 tablet_bootstrap.cc:654] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:40.473315 9404 log.cc:824] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:40.473438 9405 tablet_bootstrap.cc:654] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:40.474272 9403 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: No bootstrap required, opened a new log
I20260927 13:43:40.474627 9403 ts_tablet_manager.cc:1397] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:43:40.475350 9405 log.cc:824] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:40.477618 9404 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: No bootstrap required, opened a new log
I20260927 13:43:40.477849 9404 ts_tablet_manager.cc:1397] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 13:43:40.479440 9405 tablet_bootstrap.cc:492] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: No bootstrap required, opened a new log
I20260927 13:43:40.479669 9405 ts_tablet_manager.cc:1397] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 13:43:40.486312 9403 raft_consensus.cc:348] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.486697 9403 raft_consensus.cc:374] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:40.486788 9403 raft_consensus.cc:729] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867bbe567f7f431b88e06f660bfc65a7, State: Initialized, Role: FOLLOWER
I20260927 13:43:40.487082 9405 raft_consensus.cc:348] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.487445 9405 raft_consensus.cc:374] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:40.487535 9405 raft_consensus.cc:729] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f17f14f644f47bd427309a83ad90b, State: Initialized, Role: FOLLOWER
I20260927 13:43:40.487383 9403 consensus_queue.cc:260] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.488123 9405 consensus_queue.cc:260] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.488404 9226 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
I20260927 13:43:40.488271 9404 raft_consensus.cc:348] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.488626 9404 raft_consensus.cc:374] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:40.488708 9404 raft_consensus.cc:729] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1526686335d49e7ba21bd9cb437eb1f, State: Initialized, Role: FOLLOWER
I20260927 13:43:40.489257 9404 consensus_queue.cc:260] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.493320 9403 ts_tablet_manager.cc:1428] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Time spent starting tablet: real 0.019s user 0.012s sys 0.000s
I20260927 13:43:40.496716 9368 heartbeater.cc:503] Master 127.1.237.254:34881 was elected leader, sending a full tablet report...
I20260927 13:43:40.496868 9404 ts_tablet_manager.cc:1428] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Time spent starting tablet: real 0.019s user 0.008s sys 0.007s
I20260927 13:43:40.498492 9405 ts_tablet_manager.cc:1428] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
W20260927 13:43:40.519380 9228 tablet.cc:2369] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:43:40.603883 9370 tablet.cc:2369] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:40.631134 9409 raft_consensus.cc:482] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:40.631438 9409 raft_consensus.cc:504] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.632943 9409 leader_election.cc:288] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031), 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:40.641461 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.633293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:54865 (local address 127.1.237.195:39839)
0927 13:43:40.633484 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.633490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.634547 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 13:43:40.634718 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.634723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.634789 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.634872 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:40.635457 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.636430 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.639224 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.640377 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.640525 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.640595 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.641104 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.641218 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:43:40.641461 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.633249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:51727 (local address 127.1.237.193:33031)
0927 13:43:40.633438 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.633445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.634317 (+ 872us) server_negotiation.cc:407] Connection header received
0927 13:43:40.634535 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.634540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.634632 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.634721 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:40.635337 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.636412 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.639221 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.640390 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.640525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.640595 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.641104 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.641219 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:43:40.641723 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.633300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33031 (local address 127.1.237.194:51727)
0927 13:43:40.634217 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.634232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.634391 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.634736 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.634738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.634756 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:40.635199 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.635213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.636557 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.636561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.639066 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.639074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.640674 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.640679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.640902 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.640994 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.641068 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.641246 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":223,"thread_start_us":521,"threads_started":1}
I20260927 13:43:40.641743 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.633680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39839 (local address 127.1.237.194:54865)
0927 13:43:40.634470 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:40.634487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:40.634581 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:40.634900 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:40.634908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:40.634953 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:40.635314 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.635325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.636556 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.636560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.639060 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:40.639074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.640656 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.640666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:40.640886 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.640975 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 13:43:40.641068 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:43:40.641247 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":239,"spinlock_wait_cycles":2432,"thread_start_us":505,"threads_started":1}
I20260927 13:43:40.642597 9322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f8aa3a4c2349eaaa870dad4bc6e9bf" candidate_uuid: "867bbe567f7f431b88e06f660bfc65a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f17f14f644f47bd427309a83ad90b" is_pre_election: true
I20260927 13:43:40.642748 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f8aa3a4c2349eaaa870dad4bc6e9bf" candidate_uuid: "867bbe567f7f431b88e06f660bfc65a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1526686335d49e7ba21bd9cb437eb1f" is_pre_election: true
I20260927 13:43:40.643078 9322 raft_consensus.cc:2454] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 867bbe567f7f431b88e06f660bfc65a7 in term 0.
I20260927 13:43:40.643271 9035 raft_consensus.cc:2454] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 867bbe567f7f431b88e06f660bfc65a7 in term 0.
I20260927 13:43:40.643729 9113 leader_election.cc:302] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f17f14f644f47bd427309a83ad90b, 867bbe567f7f431b88e06f660bfc65a7; no voters:
I20260927 13:43:40.644059 9409 raft_consensus.cc:2781] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:40.644212 9409 raft_consensus.cc:482] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:40.644313 9409 raft_consensus.cc:3037] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:40.646979 9409 raft_consensus.cc:504] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.647672 9409 leader_election.cc:288] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [CANDIDATE]: Term 1 election: Requested vote from peers a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031), 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839)
I20260927 13:43:40.647954 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f8aa3a4c2349eaaa870dad4bc6e9bf" candidate_uuid: "867bbe567f7f431b88e06f660bfc65a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1526686335d49e7ba21bd9cb437eb1f"
I20260927 13:43:40.648073 9322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7f8aa3a4c2349eaaa870dad4bc6e9bf" candidate_uuid: "867bbe567f7f431b88e06f660bfc65a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4f17f14f644f47bd427309a83ad90b"
I20260927 13:43:40.648165 9035 raft_consensus.cc:3037] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:40.648288 9322 raft_consensus.cc:3037] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:40.650287 9035 raft_consensus.cc:2454] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 867bbe567f7f431b88e06f660bfc65a7 in term 1.
I20260927 13:43:40.650502 9322 raft_consensus.cc:2454] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 867bbe567f7f431b88e06f660bfc65a7 in term 1.
I20260927 13:43:40.650599 9116 leader_election.cc:302] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 867bbe567f7f431b88e06f660bfc65a7, a1526686335d49e7ba21bd9cb437eb1f; no voters:
I20260927 13:43:40.650941 9409 raft_consensus.cc:2781] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:40.651357 9409 raft_consensus.cc:686] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 1 LEADER]: Becoming Leader. State: Replica: 867bbe567f7f431b88e06f660bfc65a7, State: Running, Role: LEADER
I20260927 13:43:40.652146 9409 consensus_queue.cc:237] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:40.656455 8115 catalog_manager.cc:5495] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "867bbe567f7f431b88e06f660bfc65a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:40.686936 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.681954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.1.237.194:43445)
0927 13:43:40.682152 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.682159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.682261 (+ 102us) server_negotiation.cc:407] Connection header received
0927 13:43:40.682397 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.682400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.682474 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.682586 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:40.682872 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.683941 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.685135 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.685490 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.685574 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.685925 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:40.686414 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:40.686519 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.686605 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.686658 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:43:40.694703 9322 raft_consensus.cc:1261] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Refusing update from remote peer 867bbe567f7f431b88e06f660bfc65a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:40.694705 9035 raft_consensus.cc:1261] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Refusing update from remote peer 867bbe567f7f431b88e06f660bfc65a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:40.695740 9414 consensus_queue.cc:1035] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:40.696130 9409 consensus_queue.cc:1035] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 13:43:40.706642 9083 tablet.cc:2369] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:40.709811 9417 mvcc.cc:205] Tried to move back new op lower bound from 7333956078355738624 to 7333956078194278400. Current Snapshot: MvccSnapshot[applied={T|T < 7333956078355738624}]
I20260927 13:43:40.712011 9418 mvcc.cc:205] Tried to move back new op lower bound from 7333956078355738624 to 7333956078194278400. Current Snapshot: MvccSnapshot[applied={T|T < 7333956078355738624}]
I20260927 13:43:40.728444 8115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:40.728664 8115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:40.734408 8115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 578daa7b1b23425589168c5d6c66bf0e: Sending DeleteTablet for 3 replicas of tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf
I20260927 13:43:40.735867 9302 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:40 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:37472
I20260927 13:43:40.735925 9160 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:40 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:33320
I20260927 13:43:40.736095 9015 tablet_service.cc:1514] Processing DeleteTablet for tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:40 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:46624
I20260927 13:43:40.736774 9426 tablet_replica.cc:330] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: stopping tablet replica
I20260927 13:43:40.737223 9426 raft_consensus.cc:2229] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:40.737774 9426 raft_consensus.cc:2258] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:40.739830 9426 ts_tablet_manager.cc:1905] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:40.746711 9428 tablet_replica.cc:330] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: stopping tablet replica
I20260927 13:43:40.747144 9426 ts_tablet_manager.cc:1918] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:40.747673 9428 raft_consensus.cc:2229] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:40.747900 9426 log.cc:1196] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/f7f8aa3a4c2349eaaa870dad4bc6e9bf
I20260927 13:43:40.748860 9426 ts_tablet_manager.cc:1939] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 3c4f17f14f644f47bd427309a83ad90b: Deleting consensus metadata
I20260927 13:43:40.749312 9428 raft_consensus.cc:2258] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:40.749325 9427 tablet_replica.cc:330] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: stopping tablet replica
I20260927 13:43:40.749924 9427 raft_consensus.cc:2229] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:40.750746 9427 raft_consensus.cc:2258] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:40.751442 8099 catalog_manager.cc:4841] TS 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839): tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (table test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]) successfully deleted
I20260927 13:43:40.752713 9428 ts_tablet_manager.cc:1905] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:40.753511 9427 ts_tablet_manager.cc:1905] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:40.755352 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8083
I20260927 13:43:40.758764 8145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:40.760776 9428 ts_tablet_manager.cc:1918] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:40.760965 9428 log.cc:1196] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/f7f8aa3a4c2349eaaa870dad4bc6e9bf
I20260927 13:43:40.762045 9428 ts_tablet_manager.cc:1939] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P a1526686335d49e7ba21bd9cb437eb1f: Deleting consensus metadata
I20260927 13:43:40.764405 9427 ts_tablet_manager.cc:1918] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:40.764554 9427 log.cc:1196] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/f7f8aa3a4c2349eaaa870dad4bc6e9bf
I20260927 13:43:40.765091 9427 ts_tablet_manager.cc:1939] T f7f8aa3a4c2349eaaa870dad4bc6e9bf P 867bbe567f7f431b88e06f660bfc65a7: Deleting consensus metadata
I20260927 13:43:40.766494 8099 catalog_manager.cc:4841] TS 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445): tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (table test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]) successfully deleted
I20260927 13:43:40.778342 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.738712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.1.237.253:38091)
0927 13:43:40.738896 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.738904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.754232 (+ 15328us) server_negotiation.cc:407] Connection header received
0927 13:43:40.754600 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.754605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.754731 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.755856 (+ 1125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.756992 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.758194 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.760282 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.760882 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.761000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.772132 (+ 11132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.772138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.772142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.772176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.775600 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.776300 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.776305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.776309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.776438 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.776828 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.776833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.776837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.777130 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.777331 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.777828 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.778019 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:43:40.795881 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.754258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.1.237.252:41075)
0927 13:43:40.754406 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.754411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.779967 (+ 25556us) server_negotiation.cc:407] Connection header received
0927 13:43:40.780187 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.780192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.780263 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.780328 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.782070 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.783123 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.784850 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.785477 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.785586 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.788948 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.788954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.788959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.788994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.792338 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.793101 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.793107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.793111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.793268 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.793713 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.793718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.793722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.794075 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.794709 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.795273 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.795493 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:43:40.903759 8102 catalog_manager.cc:4841] TS a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031): tablet f7f8aa3a4c2349eaaa870dad4bc6e9bf (table test-table [id=aadb40c29e714a7e96cbb6246fb4ea43]) successfully deleted
W20260927 13:43:40.904057 8102 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.237.193:33031, user_credentials={real_user=slave}} blocked reactor thread for 139901us
I20260927 13:43:40.904559 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8083
I20260927 13:43:40.947868 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.929578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.1.237.253:38091)
0927 13:43:40.929823 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.929832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.930828 (+ 996us) server_negotiation.cc:407] Connection header received
0927 13:43:40.931113 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.931122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.931311 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.931369 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.933821 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.934914 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.936176 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.936855 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.936987 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.941357 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.941362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.941369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.941406 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.945044 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.945753 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.945757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.945759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.945864 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.946237 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.946244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.946249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.946626 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.946843 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.947196 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.947553 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 13:43:40.948448 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:40.930965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.1.237.252:41075)
0927 13:43:40.931202 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:40.931208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:40.931226 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:43:40.931283 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:40.931288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:40.931360 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:40.931473 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:40.935092 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.936152 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.937881 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:40.938586 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:40.938701 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:40.941283 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:40.941291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:40.941296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:40.941330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:40.945041 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.946332 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.946337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.946341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.946505 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:40.946985 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:40.946991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:40.946996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:40.947359 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:40.947561 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:40.947970 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 13:43:40.948167 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:43:41.715703 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:41.715021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:42023)
0927 13:43:41.715430 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:41.715546 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":242,"threads_started":1}
W20260927 13:43:41.716065 9226 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:34881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:41.721249 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:41.720492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:40165)
0927 13:43:41.720949 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:41.721064 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":225,"threads_started":1}
W20260927 13:43:41.721681 9081 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:34881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:41.724937 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:41.724185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:36299)
0927 13:43:41.724579 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:41.724706 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":218,"threads_started":1}
W20260927 13:43:41.725232 9368 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:34881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.234833 9440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 578daa7b1b23425589168c5d6c66bf0e)
I20260927 13:43:42.235181 9440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } }
I20260927 13:43:42.235570 9441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 578daa7b1b23425589168c5d6c66bf0e)
I20260927 13:43:42.235863 9441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } }
I20260927 13:43:42.237176 9440 leader_election.cc:288] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881), 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252:41075)
I20260927 13:43:42.238178 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.237261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52900)
0927 13:43:42.237734 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.237908 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"threads_started":1}
W20260927 13:43:42.242545 8699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.243290 9441 leader_election.cc:288] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881), 041a10c1b226426c96f64572df2f027f (127.1.237.253:38091)
I20260927 13:43:42.244695 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.244052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52932)
0927 13:43:42.244213 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.244417 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 13:43:42.244865 8699 leader_election.cc:334] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881): Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.251092 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.237627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52916)
0927 13:43:42.250496 (+ 12869us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.250774 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12738,"threads_started":1}
W20260927 13:43:42.251638 8888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.251775 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.237359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:41075 (local address 127.0.0.1:49742)
0927 13:43:42.238507 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.238534 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.238684 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.244377 (+ 5693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.244385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.244426 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.244780 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.244795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.247839 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.247845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.249607 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.249618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.251084 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.251090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.251289 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.251321 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.251388 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.251521 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":1964,"threads_started":1}
I20260927 13:43:42.252664 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.237914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.1.237.252:41075)
0927 13:43:42.243961 (+ 6047us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.243968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.243992 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:43:42.244071 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.244079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.244172 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.245036 (+ 864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.245278 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.246987 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.249799 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.250903 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.252039 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.252130 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.252176 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.252304 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5903,"thread_start_us":5775,"threads_started":1}
I20260927 13:43:42.253360 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "041a10c1b226426c96f64572df2f027f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5811d4ee81764b10b5b794bbaefdf35e" is_pre_election: true
I20260927 13:43:42.253698 8912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041a10c1b226426c96f64572df2f027f in term 1.
I20260927 13:43:42.254343 8698 leader_election.cc:302] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 041a10c1b226426c96f64572df2f027f, 5811d4ee81764b10b5b794bbaefdf35e; no voters: 578daa7b1b23425589168c5d6c66bf0e
I20260927 13:43:42.254624 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.253242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52934)
0927 13:43:42.254368 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.254452 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":979}
I20260927 13:43:42.254725 9440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 13:43:42.254829 9440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 578daa7b1b23425589168c5d6c66bf0e)
W20260927 13:43:42.254813 8888 leader_election.cc:334] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881): Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.254911 9440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:43:42.255362 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.241432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.1.237.253:38091)
0927 13:43:42.241942 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.241950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.247602 (+ 5652us) server_negotiation.cc:407] Connection header received
0927 13:43:42.247780 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.247788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.247885 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.248039 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.248685 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.250313 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.252999 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.254217 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.254338 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.254449 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.254946 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.255079 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":877,"threads_started":1}
I20260927 13:43:42.255762 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.237604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:38091 (local address 127.0.0.1:59274)
0927 13:43:42.247510 (+ 9906us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.247528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.247665 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.248049 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.248055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.248085 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.248482 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.248493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.250440 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.250446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.252840 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.252855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.254565 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.254571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.254774 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.254809 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.255467 (+ 658us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.255546 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9783,"thread_start_us":9629,"threads_started":1}
I20260927 13:43:42.258184 9440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } }
I20260927 13:43:42.258948 9440 leader_election.cc:288] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 election: Requested vote from peers 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881), 5811d4ee81764b10b5b794bbaefdf35e (127.1.237.252:41075)
I20260927 13:43:42.259684 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.259304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52942)
0927 13:43:42.259455 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.259534 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:43:42.259796 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5811d4ee81764b10b5b794bbaefdf35e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "041a10c1b226426c96f64572df2f027f" is_pre_election: true
I20260927 13:43:42.260126 8723 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5811d4ee81764b10b5b794bbaefdf35e in current term 2: Already voted for candidate 041a10c1b226426c96f64572df2f027f in this term.
I20260927 13:43:42.260775 8887 leader_election.cc:302] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5811d4ee81764b10b5b794bbaefdf35e; no voters: 041a10c1b226426c96f64572df2f027f, 578daa7b1b23425589168c5d6c66bf0e
I20260927 13:43:42.261142 9441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:43:42.261511 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "041a10c1b226426c96f64572df2f027f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5811d4ee81764b10b5b794bbaefdf35e"
I20260927 13:43:42.261721 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.260989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52962)
0927 13:43:42.261526 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.261609 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
W20260927 13:43:42.261886 8699 leader_election.cc:334] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881): Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
I20260927 13:43:42.263639 9441 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:43:42.265614 8912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041a10c1b226426c96f64572df2f027f in term 2.
I20260927 13:43:42.266320 8698 leader_election.cc:302] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 041a10c1b226426c96f64572df2f027f, 5811d4ee81764b10b5b794bbaefdf35e; no voters: 578daa7b1b23425589168c5d6c66bf0e
I20260927 13:43:42.266587 9440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:43:42.266979 9440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [term 2 LEADER]: Becoming Leader. State: Replica: 041a10c1b226426c96f64572df2f027f, State: Running, Role: LEADER
I20260927 13:43:42.267338 9440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } }
I20260927 13:43:42.268970 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041a10c1b226426c96f64572df2f027f. Latest consensus state: current_term: 2 leader_uuid: "041a10c1b226426c96f64572df2f027f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:42.269369 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:42.269935 9452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:42.273275 9452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:42.274052 9452 catalog_manager.cc:1259] Loaded cluster ID: 5a274b1f63aa4803a437882c40afd6df
I20260927 13:43:42.274115 9452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:42.275313 9452 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:42.276063 9452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Loaded TSK: 0
I20260927 13:43:42.276811 9452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:43:42.332937 8713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:42.344441 8713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:42.348873 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.348321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52994)
0927 13:43:42.348572 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.348690 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 13:43:42.349262 8912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [term 2 FOLLOWER]: Refusing update from remote peer 041a10c1b226426c96f64572df2f027f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 13:43:42.350099 9450 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 13:43:42.350418 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.349624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:52998)
0927 13:43:42.350122 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.350217 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260927 13:43:42.350833 8699 consensus_peers.cc:487] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f -> Peer 578daa7b1b23425589168c5d6c66bf0e (127.1.237.254:34881): Couldn't send request to peer 578daa7b1b23425589168c5d6c66bf0e. Status: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 13:43:42.353506 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041a10c1b226426c96f64572df2f027f. Latest consensus state: current_term: 2 leader_uuid: "041a10c1b226426c96f64572df2f027f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:42.353837 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:42.355412 9450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "041a10c1b226426c96f64572df2f027f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:42.355718 9450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:42.358316 9454 mvcc.cc:205] Tried to move back new op lower bound from 7333956085133422592 to 7333956084809912320. Current Snapshot: MvccSnapshot[applied={T|T < 7333956085133422592}]
I20260927 13:43:42.368786 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "041a10c1b226426c96f64572df2f027f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "578daa7b1b23425589168c5d6c66bf0e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34881 } } peers { permanent_uuid: "041a10c1b226426c96f64572df2f027f" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 38091 } attrs { promote: false } } peers { permanent_uuid: "5811d4ee81764b10b5b794bbaefdf35e" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 41075 } attrs { promote: false } } }
I20260927 13:43:42.369275 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5811d4ee81764b10b5b794bbaefdf35e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:43:42.385375 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.378986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33031 (local address 127.0.0.1:46626)
0927 13:43:42.379212 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.379230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.379345 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.380419 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.380423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.380447 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.380646 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.380661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.381914 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.381920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.383626 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.383637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.384657 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.384662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.384849 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.384871 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.384978 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.385095 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 13:43:42.385399 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.379097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.1.237.193:33031)
0927 13:43:42.379640 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.379646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.379663 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:42.379733 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.379739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.379812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.379916 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.381033 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.381772 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.383769 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.384522 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.384649 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.384750 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.385001 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.385127 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":335,"threads_started":1}
I20260927 13:43:42.386293 9015 tablet_service.cc:1467] Processing CreateTablet for tablet 09ca15833f7e4e6297d6ab425558382a (DEFAULT_TABLE table=test-table [id=86a34acfef3647beb506a1e0bac701a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:42.386708 9015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca15833f7e4e6297d6ab425558382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:42.392218 9463 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Bootstrap starting.
I20260927 13:43:42.393823 9463 tablet_bootstrap.cc:654] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:42.396060 9463 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: No bootstrap required, opened a new log
I20260927 13:43:42.396198 9463 ts_tablet_manager.cc:1397] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 13:43:42.397100 9463 raft_consensus.cc:348] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.397424 9463 raft_consensus.cc:374] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:42.397460 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.379838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:43445 (local address 127.0.0.1:33338)
0927 13:43:42.380192 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.380216 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.380300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.391864 (+ 11564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.391868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.391909 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.392132 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.392141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.393495 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.393501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.395098 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.395118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.396636 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.396642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.396797 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.396818 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.396925 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.397067 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":2}
I20260927 13:43:42.397526 9463 raft_consensus.cc:729] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1526686335d49e7ba21bd9cb437eb1f, State: Initialized, Role: FOLLOWER
I20260927 13:43:42.397799 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.380438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:39839 (local address 127.0.0.1:37482)
0927 13:43:42.385606 (+ 5168us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.385623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.385730 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.391637 (+ 5907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.391641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.391667 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.391846 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.391856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.393281 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.393286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.395282 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.395297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.396938 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.396943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.397166 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.397202 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.397323 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.397429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5051,"mutex_wait_us":24,"thread_start_us":7114,"threads_started":1}
I20260927 13:43:42.397876 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.380544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.1.237.195:39839)
0927 13:43:42.381092 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.381099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.391267 (+ 10168us) server_negotiation.cc:407] Connection header received
0927 13:43:42.391345 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.391351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.391469 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.391620 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.392039 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.393130 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.395483 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.396778 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.396920 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.397016 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.397412 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.397562 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":340,"threads_started":1}
I20260927 13:43:42.397748 9463 consensus_queue.cc:260] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.398577 9302 tablet_service.cc:1467] Processing CreateTablet for tablet 09ca15833f7e4e6297d6ab425558382a (DEFAULT_TABLE table=test-table [id=86a34acfef3647beb506a1e0bac701a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:42.398841 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.379844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.1.237.194:43445)
0927 13:43:42.391514 (+ 11670us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.391518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.391533 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:43:42.391634 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.391639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.391715 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.392269 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.392421 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.393371 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.395361 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.396444 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.396588 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.396690 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.398374 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.398530 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11571,"thread_start_us":11487,"threads_started":1}
I20260927 13:43:42.398975 9302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca15833f7e4e6297d6ab425558382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:42.399784 9160 tablet_service.cc:1467] Processing CreateTablet for tablet 09ca15833f7e4e6297d6ab425558382a (DEFAULT_TABLE table=test-table [id=86a34acfef3647beb506a1e0bac701a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:42.400177 9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ca15833f7e4e6297d6ab425558382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:42.405133 9467 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Bootstrap starting.
I20260927 13:43:42.406405 9468 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Bootstrap starting.
I20260927 13:43:42.406672 9467 tablet_bootstrap.cc:654] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:42.406985 9463 ts_tablet_manager.cc:1428] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:42.407526 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.398920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:54785)
0927 13:43:42.407079 (+ 8159us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.407278 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8013,"thread_start_us":7880,"threads_started":1}
I20260927 13:43:42.408227 9468 tablet_bootstrap.cc:654] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:42.408914 9082 heartbeater.cc:503] Master 127.1.237.253:38091 was elected leader, sending a full tablet report...
I20260927 13:43:42.409054 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.408659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:48575)
0927 13:43:42.408805 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.408888 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:43:42.410800 9468 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: No bootstrap required, opened a new log
I20260927 13:43:42.410970 9468 ts_tablet_manager.cc:1397] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 13:43:42.412041 9468 raft_consensus.cc:348] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.412330 9468 raft_consensus.cc:374] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:42.412442 9468 raft_consensus.cc:729] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f17f14f644f47bd427309a83ad90b, State: Initialized, Role: FOLLOWER
I20260927 13:43:42.412719 9468 consensus_queue.cc:260] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.413790 9468 ts_tablet_manager.cc:1428] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:42.414785 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.413850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:53329)
0927 13:43:42.414356 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.414474 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260927 13:43:42.415771 9369 heartbeater.cc:503] Master 127.1.237.253:38091 was elected leader, sending a full tablet report...
I20260927 13:43:42.417788 9467 tablet_bootstrap.cc:492] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: No bootstrap required, opened a new log
I20260927 13:43:42.417954 9467 ts_tablet_manager.cc:1397] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260927 13:43:42.418913 9467 raft_consensus.cc:348] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.419741 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.415484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:55577)
0927 13:43:42.419368 (+ 3884us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.419492 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3786,"threads_started":1}
I20260927 13:43:42.419800 9467 raft_consensus.cc:374] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:42.419948 9467 raft_consensus.cc:729] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867bbe567f7f431b88e06f660bfc65a7, State: Initialized, Role: FOLLOWER
I20260927 13:43:42.420163 9467 consensus_queue.cc:260] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.420945 9467 ts_tablet_manager.cc:1428] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 13:43:42.424039 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.421493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:53349)
0927 13:43:42.423593 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.423766 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1976,"thread_start_us":1826,"threads_started":1}
I20260927 13:43:42.425187 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.424803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:48897)
0927 13:43:42.424953 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.425036 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 13:43:42.425630 9227 heartbeater.cc:503] Master 127.1.237.253:38091 was elected leader, sending a full tablet report...
I20260927 13:43:42.452667 9471 raft_consensus.cc:482] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:42.452886 9471 raft_consensus.cc:504] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.454385 9471 leader_election.cc:288] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031), 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:42.459808 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.454702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:43445 (local address 127.1.237.195:59059)
0927 13:43:42.454927 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.454953 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.455075 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.455443 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.455447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.455467 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.455694 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.455701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.456609 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.456617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.458080 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.458091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.459114 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.459120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.459327 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.459359 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.459440 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.459547 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":82}
I20260927 13:43:42.459831 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.454712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:59059 (local address 127.1.237.194:43445)
0927 13:43:42.454880 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.454886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.455055 (+ 169us) server_negotiation.cc:407] Connection header received
0927 13:43:42.455255 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.455258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.455305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.455411 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.455820 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.456476 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.458262 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.458949 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.459070 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.459114 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.459486 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.459607 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:42.459900 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.454708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:36839 (local address 127.1.237.193:33031)
0927 13:43:42.454920 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.454926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.454949 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:43:42.455216 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.455223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.455298 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.455377 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:42.455900 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.456577 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.458237 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.459121 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.459289 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.459352 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.459585 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.459695 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:43:42.460381 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.454556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33031 (local address 127.1.237.195:36839)
0927 13:43:42.454878 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.454896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:42.455035 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:42.455414 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:42.455419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:42.455440 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:42.455747 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.455775 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.456701 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.456704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.458085 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:42.458096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.459300 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.459304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:42.459412 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.459459 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 13:43:42.459550 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 13:43:42.459655 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":43}
I20260927 13:43:42.460668 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca15833f7e4e6297d6ab425558382a" candidate_uuid: "3c4f17f14f644f47bd427309a83ad90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "867bbe567f7f431b88e06f660bfc65a7" is_pre_election: true
I20260927 13:43:42.460919 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca15833f7e4e6297d6ab425558382a" candidate_uuid: "3c4f17f14f644f47bd427309a83ad90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1526686335d49e7ba21bd9cb437eb1f" is_pre_election: true
I20260927 13:43:42.461061 9180 raft_consensus.cc:2454] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f17f14f644f47bd427309a83ad90b in term 0.
I20260927 13:43:42.461164 9035 raft_consensus.cc:2454] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4f17f14f644f47bd427309a83ad90b in term 0.
I20260927 13:43:42.461678 9258 leader_election.cc:302] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c4f17f14f644f47bd427309a83ad90b, 867bbe567f7f431b88e06f660bfc65a7, a1526686335d49e7ba21bd9cb437eb1f; no voters:
I20260927 13:43:42.462042 9471 raft_consensus.cc:2781] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:42.462164 9471 raft_consensus.cc:482] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:42.462265 9471 raft_consensus.cc:3037] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:42.464730 9471 raft_consensus.cc:504] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.465236 9471 leader_election.cc:288] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [CANDIDATE]: Term 1 election: Requested vote from peers a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031), 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445)
I20260927 13:43:42.465667 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca15833f7e4e6297d6ab425558382a" candidate_uuid: "3c4f17f14f644f47bd427309a83ad90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "867bbe567f7f431b88e06f660bfc65a7"
I20260927 13:43:42.465668 9035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ca15833f7e4e6297d6ab425558382a" candidate_uuid: "3c4f17f14f644f47bd427309a83ad90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1526686335d49e7ba21bd9cb437eb1f"
I20260927 13:43:42.465847 9180 raft_consensus.cc:3037] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:42.465857 9035 raft_consensus.cc:3037] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:42.467728 9035 raft_consensus.cc:2454] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f17f14f644f47bd427309a83ad90b in term 1.
I20260927 13:43:42.467772 9180 raft_consensus.cc:2454] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4f17f14f644f47bd427309a83ad90b in term 1.
I20260927 13:43:42.468116 9258 leader_election.cc:302] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4f17f14f644f47bd427309a83ad90b, a1526686335d49e7ba21bd9cb437eb1f; no voters:
I20260927 13:43:42.468394 9471 raft_consensus.cc:2781] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:42.468734 9471 raft_consensus.cc:686] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f17f14f644f47bd427309a83ad90b, State: Running, Role: LEADER
I20260927 13:43:42.469100 9471 consensus_queue.cc:237] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } }
I20260927 13:43:42.469549 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.469220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:34957)
0927 13:43:42.469318 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.469438 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 13:43:42.472290 8712 catalog_manager.cc:5495] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195). New cstate: current_term: 1 leader_uuid: "3c4f17f14f644f47bd427309a83ad90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4f17f14f644f47bd427309a83ad90b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 39839 } health_report { overall_health: HEALTHY } } }
I20260927 13:43:42.498067 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.493209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.1.237.195:39839)
0927 13:43:42.493381 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.493389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.493406 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:42.493625 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.493628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.493701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.493841 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:42.494172 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.494833 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.496128 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.496515 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.496636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.497010 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:42.497547 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:42.497676 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.497765 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.497818 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:43:42.501720 9035 raft_consensus.cc:1261] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Refusing update from remote peer 3c4f17f14f644f47bd427309a83ad90b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:42.502040 9180 raft_consensus.cc:1261] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Refusing update from remote peer 3c4f17f14f644f47bd427309a83ad90b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:42.502341 9476 consensus_queue.cc:1035] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1526686335d49e7ba21bd9cb437eb1f" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:42.502733 9471 consensus_queue.cc:1035] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [LEADER]: Connected to new peer: Peer: permanent_uuid: "867bbe567f7f431b88e06f660bfc65a7" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:42.505081 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.504391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:40129)
0927 13:43:42.504605 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.504838 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 13:43:42.507057 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.506303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:37841)
0927 13:43:42.506787 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.506909 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260927 13:43:42.510169 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.509784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:35295)
0927 13:43:42.509950 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.510023 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 13:43:42.510303 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.509902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.194:60671)
0927 13:43:42.510104 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.510181 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 13:43:42.520128 8713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:42.520363 8713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:42.526168 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.525733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.195:47909)
0927 13:43:42.525906 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.526013 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 13:43:42.526357 9480 mvcc.cc:205] Tried to move back new op lower bound from 7333956085760815104 to 7333956085637185536. Current Snapshot: MvccSnapshot[applied={T|T < 7333956085760815104}]
I20260927 13:43:42.528074 8713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 041a10c1b226426c96f64572df2f027f: Sending DeleteTablet for 3 replicas of tablet 09ca15833f7e4e6297d6ab425558382a
I20260927 13:43:42.528139 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.527759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.1.237.193:50291)
0927 13:43:42.527923 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.528014 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 13:43:42.528806 9015 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ca15833f7e4e6297d6ab425558382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:42 UTC) from {username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:46626
I20260927 13:43:42.529199 9160 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ca15833f7e4e6297d6ab425558382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:42 UTC) from {username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:33338
I20260927 13:43:42.529435 9302 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ca15833f7e4e6297d6ab425558382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:42 UTC) from {username='oryx', principal='oryx/127.1.237.253@KRBTEST.COM'} at 127.0.0.1:37482
I20260927 13:43:42.531955 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8874
I20260927 13:43:42.538938 9490 tablet_replica.cc:330] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: stopping tablet replica
I20260927 13:43:42.539166 9490 raft_consensus.cc:2229] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:42.539476 9490 raft_consensus.cc:2258] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:42.542169 9490 ts_tablet_manager.cc:1905] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:42.548523 9491 tablet_replica.cc:330] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: stopping tablet replica
I20260927 13:43:42.548753 9491 raft_consensus.cc:2229] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:42.554136 9490 ts_tablet_manager.cc:1918] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:42.554275 9490 log.cc:1196] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/09ca15833f7e4e6297d6ab425558382a
I20260927 13:43:42.554780 9490 ts_tablet_manager.cc:1939] T 09ca15833f7e4e6297d6ab425558382a P 867bbe567f7f431b88e06f660bfc65a7: Deleting consensus metadata
I20260927 13:43:42.556056 8697 catalog_manager.cc:4841] TS 867bbe567f7f431b88e06f660bfc65a7 (127.1.237.194:43445): tablet 09ca15833f7e4e6297d6ab425558382a (table test-table [id=86a34acfef3647beb506a1e0bac701a6]) successfully deleted
I20260927 13:43:42.556628 9491 raft_consensus.cc:2258] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:42.557170 9489 tablet_replica.cc:330] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: stopping tablet replica
I20260927 13:43:42.557337 9489 raft_consensus.cc:2229] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:42.557574 9489 raft_consensus.cc:2258] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:42.557562 9491 ts_tablet_manager.cc:1905] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:42.558347 9489 ts_tablet_manager.cc:1905] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:42.562045 9489 ts_tablet_manager.cc:1918] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:42.562173 9489 log.cc:1196] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/09ca15833f7e4e6297d6ab425558382a
I20260927 13:43:42.562602 9489 ts_tablet_manager.cc:1939] T 09ca15833f7e4e6297d6ab425558382a P a1526686335d49e7ba21bd9cb437eb1f: Deleting consensus metadata
I20260927 13:43:42.563536 8700 catalog_manager.cc:4841] TS a1526686335d49e7ba21bd9cb437eb1f (127.1.237.193:33031): tablet 09ca15833f7e4e6297d6ab425558382a (table test-table [id=86a34acfef3647beb506a1e0bac701a6]) successfully deleted
I20260927 13:43:42.563700 9491 ts_tablet_manager.cc:1918] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:42.563834 9491 log.cc:1196] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/09ca15833f7e4e6297d6ab425558382a
I20260927 13:43:42.564311 9491 ts_tablet_manager.cc:1939] T 09ca15833f7e4e6297d6ab425558382a P 3c4f17f14f644f47bd427309a83ad90b: Deleting consensus metadata
I20260927 13:43:42.565241 8697 catalog_manager.cc:4841] TS 3c4f17f14f644f47bd427309a83ad90b (127.1.237.195:39839): tablet 09ca15833f7e4e6297d6ab425558382a (table test-table [id=86a34acfef3647beb506a1e0bac701a6]) successfully deleted
I20260927 13:43:42.572031 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.532895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.1.237.193:33031)
0927 13:43:42.533041 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.533048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.552945 (+ 19897us) server_negotiation.cc:407] Connection header received
0927 13:43:42.553326 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.553331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.553401 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.553655 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:42.555498 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.556613 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.563948 (+ 7335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.564320 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.564448 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.566856 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:42.566867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:42.566881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:42.566936 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:42.569409 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.570084 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.570092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.570094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.570209 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.570598 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.570604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.570612 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.570932 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:42.571233 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.571528 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.571752 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 13:43:42.572559 9075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:42.664924 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8874
I20260927 13:43:42.688050 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9092
I20260927 13:43:42.702749 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.688847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.1.237.194:43445)
0927 13:43:42.689022 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.689028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.689069 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:43:42.689246 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.689252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.689332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.689449 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:42.690734 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.691845 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.693061 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.693677 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.693799 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.696278 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:42.696295 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:42.696317 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:42.696362 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:42.700116 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.700730 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.700739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.700742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.700908 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.701264 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.701269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.701273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.701628 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:42.701953 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.702225 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.702427 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:43:42.703431 9220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:42.833724 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.833212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:53008)
0927 13:43:42.833417 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.833531 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:43:42.834913 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.834467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34881 (local address 127.0.0.1:53016)
0927 13:43:42.834621 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:42.834737 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:43:42.851549 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9092
I20260927 13:43:42.868894 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9238
I20260927 13:43:42.884017 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:42.869614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.1.237.195:39839)
0927 13:43:42.869801 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:42.869808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:42.869854 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:43:42.870020 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:42.870025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:42.870130 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:42.870244 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:42.871495 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.872694 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.874090 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:42.874773 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:42.874903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:42.877232 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:42.877247 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:42.877267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:42.877311 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:42.881237 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.881791 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.881801 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.881805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.881968 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:42.882283 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:42.882288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:42.882291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:42.882753 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:42.883065 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:42.883373 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 13:43:42.883592 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:43:42.884739 9362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:43.008160 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9238
I20260927 13:43:43.025651 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8576
I20260927 13:43:43.026662 8743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:43.140847 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8576
I20260927 13:43:43.160950 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 8677
I20260927 13:43:43.161901 8932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:43.272478 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 8677
2026-09-27T13:43:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9130 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:43 dist-test-slave-7g3l krb5kdc[9500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:43 dist-test-slave-7g3l krb5kdc[9500](info): set up 2 sockets
Sep 27 13:43:43 dist-test-slave-7g3l krb5kdc[9500](info): commencing operation
krb5kdc: starting...
W20260927 13:43:44.327863 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:44 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab.
Entry for principal oryx/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab.
I20260927 13:43:44.518157 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43205
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab
--principal=oryx/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:44.821292 9513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:44.821548 9513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:44.821709 9513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:44.828339 9513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:44.828415 9513 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 13:43:44.828496 9513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:44.828569 9513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:44.828651 9513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:44.828730 9513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:44.837750 9513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:43205
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.254.keytab
--principal=oryx/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:44.838924 9513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:44.840190 9513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:44.851177 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:44.851130 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:44.851207 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:44.851733 9513 server_base.cc:1034] running on GCE node
I20260927 13:43:44.852497 9513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:44.853747 9513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:44.854947 9513 hybrid_clock.cc:648] HybridClock initialized: now 1790516624854883 us; error 46 us; skew 500 ppm
I20260927 13:43:44.855290 9513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:44 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516624, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:44.858067 9513 init.cc:377] Logged in from keytab as oryx/127.1.237.254@KRBTEST.COM (short username oryx)
I20260927 13:43:44.859479 9513 webserver.cc:460] Webserver started at http://127.1.237.254:40231/ using document root <none> and password file <none>
I20260927 13:43:44.860049 9513 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:44.860124 9513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:44.860364 9513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:44.863080 9513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "4ca24c07817643329748af6165849fe3"
format_stamp: "Formatted at 2026-09-27 13:43:44 on dist-test-slave-7g3l"
server_key: "c63be4e221e41bbd00fb535257e02aa9"
server_key_iv: "371fee3694b1cbbdd59796fd49a689b0"
server_key_version: "encryptionkey@0"
I20260927 13:43:44.863850 9513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "4ca24c07817643329748af6165849fe3"
format_stamp: "Formatted at 2026-09-27 13:43:44 on dist-test-slave-7g3l"
server_key: "c63be4e221e41bbd00fb535257e02aa9"
server_key_iv: "371fee3694b1cbbdd59796fd49a689b0"
server_key_version: "encryptionkey@0"
I20260927 13:43:44.868146 9513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:43:44.871040 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:44.872215 9513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:43:44.872380 9513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "4ca24c07817643329748af6165849fe3"
format_stamp: "Formatted at 2026-09-27 13:43:44 on dist-test-slave-7g3l"
server_key: "c63be4e221e41bbd00fb535257e02aa9"
server_key_iv: "371fee3694b1cbbdd59796fd49a689b0"
server_key_version: "encryptionkey@0"
I20260927 13:43:44.872570 9513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:44.889576 9513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:44.893049 9513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:44.893281 9513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:44.913348 9513 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:43205
I20260927 13:43:44.913378 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:43205 every 8 connection(s)
I20260927 13:43:44.914584 9513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:44.917966 9590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:44.924165 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9513
I20260927 13:43:44.924345 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:44.925055 1975 external_mini_cluster.cc:1428] Setting key ec11cec80bce31972ad179787dca0083
I20260927 13:43:44.928242 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Bootstrap starting.
I20260927 13:43:44.931471 9590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:44.932453 9590 log.cc:824] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:44.935070 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: No bootstrap required, opened a new log
Sep 27 13:43:44 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:44.943528 9590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } }
I20260927 13:43:44.943848 9590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:44.943957 9590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca24c07817643329748af6165849fe3, State: Initialized, Role: FOLLOWER
I20260927 13:43:44.944454 9590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } }
I20260927 13:43:44.944670 9590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:44.944796 9590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:44.944974 9590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:44.946245 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:44.926892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.1.237.254:43205)
0927 13:43:44.927670 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:44.927681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:44.927718 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:43:44.929394 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:44.929416 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:44.929754 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:44.930051 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:44.931696 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:44.933010 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:44.934598 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:44.935263 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:44.935398 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:44.939811 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:44.939818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:44.939829 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:44.939864 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:44.942532 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:44.943194 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:44.943205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:44.943211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:44.943350 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:44.943704 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:44.943711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:44.943715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:44.944354 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:44.944638 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:44.944971 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:43:44.945324 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":400,"threads_started":1}
I20260927 13:43:44.947242 9590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } }
I20260927 13:43:44.947767 9590 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca24c07817643329748af6165849fe3; no voters:
I20260927 13:43:44.948287 9590 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:44.948411 9595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:44.949613 9595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca24c07817643329748af6165849fe3, State: Running, Role: LEADER
I20260927 13:43:44.950076 9590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:44.950013 9595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } }
I20260927 13:43:44.952791 9596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca24c07817643329748af6165849fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } } }
I20260927 13:43:44.952904 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca24c07817643329748af6165849fe3. Latest consensus state: current_term: 1 leader_uuid: "4ca24c07817643329748af6165849fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca24c07817643329748af6165849fe3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43205 } } }
I20260927 13:43:44.953214 9596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:44.953277 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3 [sys.catalog]: This master's current role is: LEADER
W20260927 13:43:44.962225 9611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:43:44.962325 9611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:43:44.962464 9603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:44.964568 9603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:44.972230 9603 catalog_manager.cc:1347] Generated new cluster ID: 22bfd0b09b284b8192398fa6370a4255
I20260927 13:43:44.972323 9603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:45.010263 9603 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:45.012084 9603 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:45.021379 9603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Generated new TSK 0
I20260927 13:43:45.022110 9603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
I20260927 13:43:45.068218 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43205
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:45.370790 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:45.371024 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:45.371205 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:45.378192 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:45.378279 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:45.378410 9616 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:45.378530 9616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:45.387497 9616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:45.388548 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:45.389705 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:45.401650 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:45.401904 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:45.402194 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:45.402663 9616 server_base.cc:1034] running on GCE node
I20260927 13:43:45.403208 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:45.403956 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:45.405151 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790516625405111 us; error 27 us; skew 500 ppm
I20260927 13:43:45.405398 9616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:45 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:45.408314 9616 init.cc:377] Logged in from keytab as oryx/127.1.237.193@KRBTEST.COM (short username oryx)
I20260927 13:43:45.409489 9616 webserver.cc:460] Webserver started at http://127.1.237.193:46121/ using document root <none> and password file <none>
I20260927 13:43:45.410096 9616 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:45.410198 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:45.410493 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:45.413136 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "06167796f3d74a9a9f45c88b88d78031"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "06f44a7b677c70d23d7098f6c718079b"
server_key_iv: "6d6376dc3222ffff1e170615d926accb"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.413856 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "06167796f3d74a9a9f45c88b88d78031"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "06f44a7b677c70d23d7098f6c718079b"
server_key_iv: "6d6376dc3222ffff1e170615d926accb"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.418299 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:43:45.421365 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:45.422508 9616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:45.422657 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "06167796f3d74a9a9f45c88b88d78031"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "06f44a7b677c70d23d7098f6c718079b"
server_key_iv: "6d6376dc3222ffff1e170615d926accb"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.422863 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:45.437873 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:45.442907 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:45.443218 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:45.445786 9616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:45.445926 9616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 13:43:45.446074 9616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:45.446151 9616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:43:45 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:45.468015 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.446116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:43415 (local address 127.1.237.254:43205)
0927 13:43:45.446280 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:45.446286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:45.447942 (+ 1656us) server_negotiation.cc:407] Connection header received
0927 13:43:45.450116 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:45.450120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:45.450208 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:45.450289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:45.452725 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.453983 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.456625 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.457512 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.457677 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.461856 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:45.461861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:45.461872 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:45.461901 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:45.464384 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.465175 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.465181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.465185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.465315 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.465850 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.465855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.465858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.466171 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:45.466321 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:45.467370 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 13:43:45.467618 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:45.469692 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.446944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.193:43415)
0927 13:43:45.447770 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:45.447817 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:45.449803 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:45.450876 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:45.450889 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:45.451696 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:45.452529 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.452570 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.454184 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.454192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.456415 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.456449 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.457716 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.457721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.457870 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.458517 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:45.458549 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:45.461652 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:45.464516 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.464524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.464540 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.465052 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.465438 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.465443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.465448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.465707 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.466327 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:45.466339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:45.466829 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 13:43:45.467263 (+ 434us) client_negotiation.cc:240] Negotiation successful
0927 13:43:45.467886 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":369,"threads_started":1}
I20260927 13:43:45.488349 9616 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:35227
I20260927 13:43:45.488435 9750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:35227 every 8 connection(s)
I20260927 13:43:45.489768 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.476684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:54467 (local address 127.1.237.254:43205)
0927 13:43:45.476829 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:45.476834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:45.477522 (+ 688us) server_negotiation.cc:407] Connection header received
0927 13:43:45.477683 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:45.477690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:45.477781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:45.477891 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:45.478686 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.479920 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.481431 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.482163 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.482297 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.484480 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:45.484487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:45.484493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:45.484531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:45.487162 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.488124 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.488132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.488136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.488295 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.488709 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.488712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.488714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.488927 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:45.489029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:45.489433 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 13:43:45.489574 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:43:45.489758 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:45.490075 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.476802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.193:54467)
0927 13:43:45.477401 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:45.477424 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:45.477561 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:45.477931 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:45.477937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:45.478310 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:45.478558 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.478566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.480037 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.480040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.481243 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.481277 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.482358 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.482366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.482589 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.483220 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:45.483248 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:45.484230 (+ 982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:45.487327 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.487331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.487335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.488002 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.488424 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.488428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.488433 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.488612 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.489038 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:45.489043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:45.489213 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 13:43:45.489362 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 13:43:45.489644 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":324,"threads_started":1}
I20260927 13:43:45.492067 9751 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43205
I20260927 13:43:45.492276 9751 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:45.493022 9751 heartbeater.cc:511] Master 127.1.237.254:43205 requested a full tablet report, sending...
I20260927 13:43:45.494455 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9616
I20260927 13:43:45.494657 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:45.495119 9545 ts_manager.cc:194] Registered new tserver with Master: 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227)
I20260927 13:43:45.495527 1975 external_mini_cluster.cc:1428] Setting key 2cde60514d565af8175ab2dced322db1
I20260927 13:43:45.496832 9545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:43415
WARNING: no policy specified for oryx/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
I20260927 13:43:45.534060 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43205
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:45.830086 9755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:45.830368 9755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:45.830540 9755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:45.837087 9755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:45.837203 9755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:45.837337 9755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:45.837459 9755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:45.846046 9755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:45.847139 9755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:45.848407 9755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:45.860702 9760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:45.860736 9763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:45.860776 9761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:45.861737 9755 server_base.cc:1034] running on GCE node
I20260927 13:43:45.862260 9755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:45.863021 9755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:45.864216 9755 hybrid_clock.cc:648] HybridClock initialized: now 1790516625864165 us; error 43 us; skew 500 ppm
I20260927 13:43:45.864508 9755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:45 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:45.867448 9755 init.cc:377] Logged in from keytab as oryx/127.1.237.194@KRBTEST.COM (short username oryx)
I20260927 13:43:45.868721 9755 webserver.cc:460] Webserver started at http://127.1.237.194:45123/ using document root <none> and password file <none>
I20260927 13:43:45.869287 9755 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:45.869400 9755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:45.869696 9755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:45.872331 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "cf5c81f81ed54a35bc12a0224401d9f5"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "7273d224e4ff5206d1c7fdc705200628"
server_key_iv: "a5d751f70cade48d8d28b0359e995b04"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.873070 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "cf5c81f81ed54a35bc12a0224401d9f5"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "7273d224e4ff5206d1c7fdc705200628"
server_key_iv: "a5d751f70cade48d8d28b0359e995b04"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.877562 9755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 13:43:45.880606 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:45.881841 9755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:45.882010 9755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "cf5c81f81ed54a35bc12a0224401d9f5"
format_stamp: "Formatted at 2026-09-27 13:43:45 on dist-test-slave-7g3l"
server_key: "7273d224e4ff5206d1c7fdc705200628"
server_key_iv: "a5d751f70cade48d8d28b0359e995b04"
server_key_version: "encryptionkey@0"
I20260927 13:43:45.882217 9755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:45.899029 9755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:45.902500 9755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:45.902756 9755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:45.905025 9755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:45.905119 9755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:45.905241 9755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:45.905308 9755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:43:45 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:45.929159 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.905566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43745 (local address 127.1.237.254:43205)
0927 13:43:45.905776 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:45.905785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:45.907285 (+ 1500us) server_negotiation.cc:407] Connection header received
0927 13:43:45.909235 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:45.909240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:45.909318 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:45.909431 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:45.911434 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.912431 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.917344 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.917989 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.918125 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.923294 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:45.923301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:45.923309 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:45.923355 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:45.926120 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.926841 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.926848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.926855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.927032 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.927549 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.927554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.927557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.927825 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:45.927979 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:45.928612 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 13:43:45.928833 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:43:45.930054 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.906399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.194:43745)
0927 13:43:45.907108 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:45.907140 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:45.908986 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:45.909645 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:45.909652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:45.910337 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:45.911205 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.911235 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.912607 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.912613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.917148 (+ 4535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.917165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.918272 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.918279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.918517 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.919229 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:45.919260 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:45.922949 (+ 3689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:45.926243 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.926251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.926264 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.926723 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.927136 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.927142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.927145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.927438 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.927949 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:45.927956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:45.928262 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 13:43:45.928503 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 13:43:45.928914 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260927 13:43:45.952939 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.939381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:35659 (local address 127.1.237.254:43205)
0927 13:43:45.939650 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:45.939655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:45.940410 (+ 755us) server_negotiation.cc:407] Connection header received
0927 13:43:45.940601 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:45.940608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:45.940688 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:45.940781 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:45.941663 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.942720 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.944253 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.944650 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.944789 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.946404 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:45.946409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:45.946414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:45.946449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:45.950084 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.951012 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.951019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.951024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.951165 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:45.951528 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:45.951534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:45.951537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:45.951790 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:45.951989 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:45.952355 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 13:43:45.952570 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260927 13:43:45.953032 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:45.939768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.194:35659)
0927 13:43:45.940307 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:45.940322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:45.940461 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:45.940914 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:45.940918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:45.941273 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:45.941516 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.941524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.942861 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.942866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.944115 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:45.944129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:45.944822 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:45.944827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:45.944989 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:45.945544 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:45.945558 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:45.946246 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:45.950212 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.950219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.950224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.950800 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.951283 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:45.951288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:45.951291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:45.951428 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:45.951960 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:45.951964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:45.952127 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 13:43:45.952274 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 13:43:45.952578 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
I20260927 13:43:45.967064 9755 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:42799
I20260927 13:43:45.967090 9889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:42799 every 8 connection(s)
I20260927 13:43:45.968261 9755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:45.970270 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9755
I20260927 13:43:45.970474 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:45.970480 9890 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43205
I20260927 13:43:45.970682 9890 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:45.971024 1975 external_mini_cluster.cc:1428] Setting key 5859f80eced5782cfbedd7ed2f0a2c02
I20260927 13:43:45.971467 9890 heartbeater.cc:511] Master 127.1.237.254:43205 requested a full tablet report, sending...
I20260927 13:43:45.972816 9545 ts_manager.cc:194] Registered new tserver with Master: cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799)
I20260927 13:43:45.973640 9545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:43745
WARNING: no policy specified for oryx/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
I20260927 13:43:46.016431 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43205
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:46.318750 9894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:46.318979 9894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:46.319062 9894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:46.325889 9894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:46.326016 9894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:46.326175 9894 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 13:43:46.326288 9894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:46.335214 9894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.237.254:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:46.336199 9894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:46.337383 9894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:46.348981 9899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:46.349082 9900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:46.349022 9902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:46.349625 9894 server_base.cc:1034] running on GCE node
I20260927 13:43:46.350162 9894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:46.350975 9894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:46.352217 9894 hybrid_clock.cc:648] HybridClock initialized: now 1790516626352143 us; error 39 us; skew 500 ppm
I20260927 13:43:46.352496 9894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:46 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516626, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:46.355366 9894 init.cc:377] Logged in from keytab as oryx/127.1.237.195@KRBTEST.COM (short username oryx)
I20260927 13:43:46.356649 9894 webserver.cc:460] Webserver started at http://127.1.237.195:46429/ using document root <none> and password file <none>
I20260927 13:43:46.357295 9894 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:46.357407 9894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:46.357702 9894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:46.360244 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "f1e34b5e248247b098d7926d16d4cd0e"
format_stamp: "Formatted at 2026-09-27 13:43:46 on dist-test-slave-7g3l"
server_key: "6d060292cf2d9515e429e988c98d1db7"
server_key_iv: "543f79abbd07fb763d3a8010da0f37d6"
server_key_version: "encryptionkey@0"
I20260927 13:43:46.360989 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e34b5e248247b098d7926d16d4cd0e"
format_stamp: "Formatted at 2026-09-27 13:43:46 on dist-test-slave-7g3l"
server_key: "6d060292cf2d9515e429e988c98d1db7"
server_key_iv: "543f79abbd07fb763d3a8010da0f37d6"
server_key_version: "encryptionkey@0"
I20260927 13:43:46.365716 9894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:43:46.368886 9909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:46.369966 9894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:46.370117 9894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "f1e34b5e248247b098d7926d16d4cd0e"
format_stamp: "Formatted at 2026-09-27 13:43:46 on dist-test-slave-7g3l"
server_key: "6d060292cf2d9515e429e988c98d1db7"
server_key_iv: "543f79abbd07fb763d3a8010da0f37d6"
server_key_version: "encryptionkey@0"
I20260927 13:43:46.370301 9894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:46.388381 9894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:46.391839 9894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:46.392138 9894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:46.394644 9894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:46.394743 9894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:46.394908 9894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:46.394968 9894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:43:46 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516626, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:46.415246 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:46.395048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45739 (local address 127.1.237.254:43205)
0927 13:43:46.395260 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:46.395269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:46.396628 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 13:43:46.398490 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:46.398496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:46.398584 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:46.398696 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:46.400737 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.402036 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.404250 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.404982 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.405108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:46.409582 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:46.409590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:46.409598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:46.409629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:46.412179 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:46.412834 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:46.412839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:46.412849 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:46.412972 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:46.413393 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:46.413398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:46.413401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:46.413654 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:46.413806 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:46.414676 (+ 870us) server_negotiation.cc:299] Negotiation successful
0927 13:43:46.414889 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 13:43:46.416643 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:46.395715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.195:45739)
0927 13:43:46.396475 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:46.396515 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:46.398235 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:46.398898 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:46.398904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:46.399637 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:46.400506 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:46.400535 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.402208 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.402214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:46.404108 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:46.404120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.405205 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.405212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:46.405429 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:46.406159 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:46.406204 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:46.409348 (+ 3144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:46.412304 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:46.412312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:46.412327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:46.412711 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:46.413076 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:46.413082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:46.413085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:46.413260 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:46.413782 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:46.413789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:46.414221 (+ 432us) client_negotiation.cc:769] Sending connection context
0927 13:43:46.414600 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 13:43:46.415138 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":347,"threads_started":1}
I20260927 13:43:46.439623 9894 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:44097
I20260927 13:43:46.439641 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:44097 every 8 connection(s)
I20260927 13:43:46.441102 9894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:46.442019 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:46.425611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:44417 (local address 127.1.237.254:43205)
0927 13:43:46.425784 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:46.425791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:46.426771 (+ 980us) server_negotiation.cc:407] Connection header received
0927 13:43:46.426872 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:46.426877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:46.426965 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:46.427072 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:46.428593 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.429830 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.431766 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.432563 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.432749 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:46.435821 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:46.435830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:46.435837 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:46.435878 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:46.439344 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:46.440139 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:46.440142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:46.440145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:46.440279 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:46.440665 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:46.440668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:46.440670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:46.440881 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:46.440992 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:46.441417 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 13:43:46.441595 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 13:43:46.442489 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:46.425682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.1.237.195:44417)
0927 13:43:46.426660 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:46.426679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:46.426778 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:46.427176 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:46.427182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:46.427875 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:46.428395 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:46.428414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.429990 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.429996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:46.431623 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:46.431637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.433026 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.433035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:46.433308 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:46.434147 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:46.434173 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:46.435635 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:46.439484 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:46.439489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:46.439496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:46.440003 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:46.440388 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:46.440393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:46.440396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:46.440553 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:46.440992 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:46.441000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:46.441195 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 13:43:46.441332 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 13:43:46.441721 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":496,"threads_started":1}
I20260927 13:43:46.443388 10029 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43205
I20260927 13:43:46.443621 10029 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:46.444473 10029 heartbeater.cc:511] Master 127.1.237.254:43205 requested a full tablet report, sending...
I20260927 13:43:46.445574 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 9894
I20260927 13:43:46.445753 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:46.446177 1975 external_mini_cluster.cc:1428] Setting key 472c28b8e507bf3fce03c3a2e3a7379d
I20260927 13:43:46.447525 9544 ts_manager.cc:194] Registered new tserver with Master: f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:46.448447 9544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:45739
I20260927 13:43:46.449067 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 13:43:46 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 13:43:46 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:46 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:46.499253 9751 heartbeater.cc:503] Master 127.1.237.254:43205 was elected leader, sending a full tablet report...
I20260927 13:43:46.975831 9890 heartbeater.cc:503] Master 127.1.237.254:43205 was elected leader, sending a full tablet report...
I20260927 13:43:47.450722 10029 heartbeater.cc:503] Master 127.1.237.254:43205 was elected leader, sending a full tablet report...
W20260927 13:43:47.528069 10037 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.039s user 0.007s sys 0.007s
W20260927 13:43:47.529167 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:46.483452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.1.237.254:43205)
0927 13:43:46.483676 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:46.483684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:46.483935 (+ 251us) server_negotiation.cc:407] Connection header received
0927 13:43:46.484133 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:46.484145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:46.484208 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:46.484302 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:46.485547 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.486308 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.487567 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:46.487988 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:46.488113 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.528946 (+1040833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58048: BlockingRecv error: recv got EOF from 127.0.0.1:58048 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 13:43:47 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:47.548959 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.532774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.1.237.254:43205)
0927 13:43:47.532937 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.532941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.533349 (+ 408us) server_negotiation.cc:407] Connection header received
0927 13:43:47.533494 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.533498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.533567 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.533649 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:47.534767 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.535553 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.536686 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.537046 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.537129 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.544378 (+ 7249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:47.544382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:47.544386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:47.544429 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:47.546772 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:47.547416 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:47.547419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:47.547422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:47.547542 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:47.547830 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:47.547833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:47.547835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:47.548068 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:47.548232 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.548461 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.548618 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:43:47.552493 9544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:47.558987 9544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:47.591269 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.569648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.1.237.195:44097)
0927 13:43:47.582066 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.582074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.582106 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:43:47.582228 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.582239 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.582533 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.583632 (+ 1099us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:47.583862 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.585627 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.588693 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.590159 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.590298 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.590489 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.590803 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.590967 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12246,"thread_start_us":11130,"threads_started":1}
I20260927 13:43:47.591338 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.569584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:44097 (local address 127.0.0.1:47578)
0927 13:43:47.579846 (+ 10262us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:47.579892 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:47.580087 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:47.582797 (+ 2710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:47.582808 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:47.582848 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:47.583403 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.583420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.586215 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.586220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.588538 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.588551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.590366 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.590374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.590552 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.590598 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 13:43:47.590728 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 13:43:47.590909 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9944,"mutex_wait_us":2,"thread_start_us":16214,"threads_started":1}
I20260927 13:43:47.594832 9959 tablet_service.cc:1467] Processing CreateTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 (DEFAULT_TABLE table=test-table [id=fda0f71ea5354d2998ffbc5057573b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.596051 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.581109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35227 (local address 127.0.0.1:43480)
0927 13:43:47.586085 (+ 4976us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:47.586101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:47.586201 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:47.586964 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:47.586967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:47.586987 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:47.587734 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.587758 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.589154 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.589161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.590732 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.590745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.595283 (+ 4538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.595290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.595409 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.595476 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 13:43:47.595590 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 13:43:47.595702 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4872,"mutex_wait_us":40,"thread_start_us":4381,"threads_started":1}
I20260927 13:43:47.596055 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.580536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.1.237.193:35227)
0927 13:43:47.585392 (+ 4856us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.585399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.586304 (+ 905us) server_negotiation.cc:407] Connection header received
0927 13:43:47.586425 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.586440 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.586784 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.586968 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:47.587927 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.589030 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.590890 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.592322 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.592443 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.592637 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.595634 (+ 2997us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.595772 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4726,"thread_start_us":4599,"threads_started":1}
I20260927 13:43:47.596171 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.569892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.1.237.194:42799)
0927 13:43:47.580654 (+ 10762us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.580665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.585985 (+ 5320us) server_negotiation.cc:407] Connection header received
0927 13:43:47.586299 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.586306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.586553 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.586714 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:47.587862 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.589125 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.591628 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.592976 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.593082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.593196 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.595732 (+ 2536us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.595899 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10572,"thread_start_us":11321,"threads_started":1}
I20260927 13:43:47.596247 9959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cfdb663cd94a2cbe800c879b50cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.596796 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.569581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:42799 (local address 127.0.0.1:59780)
0927 13:43:47.585827 (+ 16246us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:47.585843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:47.586127 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:47.586898 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:47.586930 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:47.586963 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:47.587667 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.587684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.589243 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.589251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.591295 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.591310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.595331 (+ 4021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.595340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.595538 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.595568 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:43:47.596300 (+ 732us) client_negotiation.cc:240] Negotiation successful
0927 13:43:47.596415 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16090,"mutex_wait_us":473,"thread_start_us":9287,"threads_started":1}
I20260927 13:43:47.599732 9681 tablet_service.cc:1467] Processing CreateTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 (DEFAULT_TABLE table=test-table [id=fda0f71ea5354d2998ffbc5057573b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.599732 9824 tablet_service.cc:1467] Processing CreateTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 (DEFAULT_TABLE table=test-table [id=fda0f71ea5354d2998ffbc5057573b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.600901 9824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cfdb663cd94a2cbe800c879b50cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.600907 9681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cfdb663cd94a2cbe800c879b50cba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.606668 10054 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Bootstrap starting.
I20260927 13:43:47.610055 10054 tablet_bootstrap.cc:654] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.610971 10054 log.cc:824] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:47.613297 10054 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: No bootstrap required, opened a new log
I20260927 13:43:47.613590 10054 ts_tablet_manager.cc:1397] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 13:43:47.614076 10056 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Bootstrap starting.
I20260927 13:43:47.614728 10057 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Bootstrap starting.
I20260927 13:43:47.617244 10056 tablet_bootstrap.cc:654] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.617882 10057 tablet_bootstrap.cc:654] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.618489 10056 log.cc:824] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:47.619051 10057 log.cc:824] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:47.621183 10057 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: No bootstrap required, opened a new log
I20260927 13:43:47.621187 10056 tablet_bootstrap.cc:492] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: No bootstrap required, opened a new log
I20260927 13:43:47.621419 10057 ts_tablet_manager.cc:1397] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 13:43:47.621425 10056 ts_tablet_manager.cc:1397] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 13:43:47.623571 10054 raft_consensus.cc:348] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.623968 10054 raft_consensus.cc:374] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.624102 10054 raft_consensus.cc:729] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.624675 10054 consensus_queue.cc:260] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.626293 10054 ts_tablet_manager.cc:1428] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260927 13:43:47.629630 10057 raft_consensus.cc:348] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.630043 10057 raft_consensus.cc:374] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.629827 10056 raft_consensus.cc:348] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.630273 10056 raft_consensus.cc:374] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.630381 10056 raft_consensus.cc:729] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.630172 10057 raft_consensus.cc:729] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.631021 10056 consensus_queue.cc:260] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.631050 10057 consensus_queue.cc:260] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.632661 10056 ts_tablet_manager.cc:1428] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 13:43:47.632740 10057 ts_tablet_manager.cc:1428] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260927 13:43:47.699048 10030 tablet.cc:2369] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:43:47.720815 9891 tablet.cc:2369] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:43:47.742961 9752 tablet.cc:2369] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:43:47.781829 10061 raft_consensus.cc:482] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:47.782112 10061 raft_consensus.cc:504] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.783715 10061 leader_election.cc:288] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097), 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227)
I20260927 13:43:47.790474 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.784172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:37699 (local address 127.1.237.193:35227)
0927 13:43:47.784385 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.784394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.785360 (+ 966us) server_negotiation.cc:407] Connection header received
0927 13:43:47.785573 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.785583 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.785646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.785737 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:47.786454 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.787132 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.788840 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.789586 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.789763 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.789837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.790120 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.790248 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 13:43:47.790510 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.784293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:44389 (local address 127.1.237.195:44097)
0927 13:43:47.784504 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.784513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.785263 (+ 750us) server_negotiation.cc:407] Connection header received
0927 13:43:47.785484 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.785488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.785556 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.785657 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:47.786282 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.787017 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.788852 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.789599 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.789778 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.789856 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.790138 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.790271 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:43:47.790634 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.784184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35227 (local address 127.1.237.194:37699)
0927 13:43:47.785276 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:47.785292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:47.785390 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:47.785755 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:47.785759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:47.785793 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:47.786264 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.786276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.787287 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.787292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.788701 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.788712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.789798 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.789804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.789966 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.789993 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 13:43:47.790096 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:43:47.790225 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":26,"thread_start_us":543,"threads_started":1}
I20260927 13:43:47.790747 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.784131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:44097 (local address 127.1.237.194:44389)
0927 13:43:47.785153 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:47.785172 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:47.785323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:47.785673 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:47.785677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:47.785703 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:47.786125 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.786136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.787171 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.787179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.788697 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:47.788708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.789841 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.789847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:47.789987 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.790008 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:43:47.790112 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:43:47.790239 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":6,"thread_start_us":755,"threads_started":1}
I20260927 13:43:47.791821 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cfdb663cd94a2cbe800c879b50cba5" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e" is_pre_election: true
I20260927 13:43:47.791821 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cfdb663cd94a2cbe800c879b50cba5" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031" is_pre_election: true
I20260927 13:43:47.792398 9701 raft_consensus.cc:2454] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:47.792398 9983 raft_consensus.cc:2454] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:47.793048 9772 leader_election.cc:302] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, cf5c81f81ed54a35bc12a0224401d9f5; no voters:
I20260927 13:43:47.793434 10061 raft_consensus.cc:2781] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:47.793586 10061 raft_consensus.cc:482] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:47.793704 10061 raft_consensus.cc:3037] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.795876 10061 raft_consensus.cc:504] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.796422 10061 leader_election.cc:288] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Requested vote from peers f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097), 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227)
I20260927 13:43:47.796862 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cfdb663cd94a2cbe800c879b50cba5" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031"
I20260927 13:43:47.796862 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cfdb663cd94a2cbe800c879b50cba5" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e"
I20260927 13:43:47.797058 9701 raft_consensus.cc:3037] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.797058 9983 raft_consensus.cc:3037] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.800191 9983 raft_consensus.cc:2454] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:47.800191 9701 raft_consensus.cc:2454] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:47.800575 9773 leader_election.cc:302] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf5c81f81ed54a35bc12a0224401d9f5, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:47.800895 10061 raft_consensus.cc:2781] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:47.801381 10061 raft_consensus.cc:686] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Becoming Leader. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Running, Role: LEADER
I20260927 13:43:47.801911 10061 consensus_queue.cc:237] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } }
I20260927 13:43:47.806363 9545 catalog_manager.cc:5495] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:47.870357 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:47.865658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.1.237.194:42799)
0927 13:43:47.865814 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:47.865819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:47.865898 (+ 79us) server_negotiation.cc:407] Connection header received
0927 13:43:47.866078 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:47.866090 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:47.866175 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:47.866302 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:47.866612 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.867367 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.868599 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:47.868963 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:47.869057 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:47.869417 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:47.869919 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:47.870024 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:47.870109 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 13:43:47.870165 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 13:43:47.878187 9701 raft_consensus.cc:1261] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:47.878187 9983 raft_consensus.cc:1261] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:47.879101 10061 consensus_queue.cc:1035] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:47.879482 10065 consensus_queue.cc:1035] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:47.905900 9544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:47.906188 9544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:47.909440 9544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Sending DeleteTablet for 3 replicas of tablet 32cfdb663cd94a2cbe800c879b50cba5
I20260927 13:43:47.910403 9959 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:47 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:47578
I20260927 13:43:47.910487 9824 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:47 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:59780
I20260927 13:43:47.910594 9681 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cfdb663cd94a2cbe800c879b50cba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:47 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43480
I20260927 13:43:47.912207 9545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:47.912775 9545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:47.923988 9958 tablet_service.cc:1467] Processing CreateTablet for tablet be1199820b0b43a9b700227c077bf362 (DEFAULT_TABLE table=test-table [id=8d64973b60144522924cd05240c8d142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.924716 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1199820b0b43a9b700227c077bf362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.926389 10078 tablet_replica.cc:330] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: stopping tablet replica
I20260927 13:43:47.927367 10078 raft_consensus.cc:2229] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:47.928476 10078 raft_consensus.cc:2258] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:47.928539 9680 tablet_service.cc:1467] Processing CreateTablet for tablet be1199820b0b43a9b700227c077bf362 (DEFAULT_TABLE table=test-table [id=8d64973b60144522924cd05240c8d142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.929198 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1199820b0b43a9b700227c077bf362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.932099 9823 tablet_service.cc:1467] Processing CreateTablet for tablet be1199820b0b43a9b700227c077bf362 (DEFAULT_TABLE table=test-table [id=8d64973b60144522924cd05240c8d142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:47.932660 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1199820b0b43a9b700227c077bf362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:47.933156 10079 tablet_replica.cc:330] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: stopping tablet replica
I20260927 13:43:47.932454 10078 ts_tablet_manager.cc:1905] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:47.939949 10056 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Bootstrap starting.
I20260927 13:43:47.939941 10079 raft_consensus.cc:2229] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:47.940367 10079 raft_consensus.cc:2258] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:47.941552 10056 tablet_bootstrap.cc:654] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.942963 10054 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Bootstrap starting.
I20260927 13:43:47.943084 10057 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Bootstrap starting.
I20260927 13:43:47.944671 10057 tablet_bootstrap.cc:654] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.945001 10054 tablet_bootstrap.cc:654] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:47.948716 10078 ts_tablet_manager.cc:1918] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:47.948845 10078 log.cc:1196] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/32cfdb663cd94a2cbe800c879b50cba5
I20260927 13:43:47.949306 10078 ts_tablet_manager.cc:1939] T 32cfdb663cd94a2cbe800c879b50cba5 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting consensus metadata
I20260927 13:43:47.952653 10077 tablet_replica.cc:330] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: stopping tablet replica
I20260927 13:43:47.953117 10054 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: No bootstrap required, opened a new log
I20260927 13:43:47.953181 10077 raft_consensus.cc:2229] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:47.953248 10054 ts_tablet_manager.cc:1397] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 13:43:47.953646 10077 raft_consensus.cc:2258] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:47.953966 10054 raft_consensus.cc:348] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.954154 10054 raft_consensus.cc:374] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.954239 10054 raft_consensus.cc:729] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.954442 10054 consensus_queue.cc:260] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.954921 10054 ts_tablet_manager.cc:1428] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:47.956333 10079 ts_tablet_manager.cc:1905] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:47.956717 10077 ts_tablet_manager.cc:1905] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:47.959398 10057 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: No bootstrap required, opened a new log
I20260927 13:43:47.959542 10057 ts_tablet_manager.cc:1397] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260927 13:43:47.959743 10056 tablet_bootstrap.cc:492] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: No bootstrap required, opened a new log
I20260927 13:43:47.959877 10056 ts_tablet_manager.cc:1397] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260927 13:43:47.960366 10057 raft_consensus.cc:348] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.960583 10057 raft_consensus.cc:374] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.960681 10057 raft_consensus.cc:729] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.960664 10056 raft_consensus.cc:348] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.960976 10056 raft_consensus.cc:374] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:47.961071 10056 raft_consensus.cc:729] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Initialized, Role: FOLLOWER
I20260927 13:43:47.960961 10057 consensus_queue.cc:260] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.961297 10056 consensus_queue.cc:260] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.961457 10057 ts_tablet_manager.cc:1428] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:47.962942 9530 catalog_manager.cc:4841] TS cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799): tablet 32cfdb663cd94a2cbe800c879b50cba5 (table test-table [id=fda0f71ea5354d2998ffbc5057573b8a]) successfully deleted
I20260927 13:43:47.963131 10077 ts_tablet_manager.cc:1918] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:47.963289 10077 log.cc:1196] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/32cfdb663cd94a2cbe800c879b50cba5
I20260927 13:43:47.963608 10056 ts_tablet_manager.cc:1428] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 13:43:47.963817 10077 ts_tablet_manager.cc:1939] T 32cfdb663cd94a2cbe800c879b50cba5 P f1e34b5e248247b098d7926d16d4cd0e: Deleting consensus metadata
I20260927 13:43:47.965550 9531 catalog_manager.cc:4841] TS f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097): tablet 32cfdb663cd94a2cbe800c879b50cba5 (table test-table [id=fda0f71ea5354d2998ffbc5057573b8a]) successfully deleted
I20260927 13:43:47.968780 10079 ts_tablet_manager.cc:1918] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:47.968981 10079 log.cc:1196] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/32cfdb663cd94a2cbe800c879b50cba5
I20260927 13:43:47.969534 10079 ts_tablet_manager.cc:1939] T 32cfdb663cd94a2cbe800c879b50cba5 P 06167796f3d74a9a9f45c88b88d78031: Deleting consensus metadata
I20260927 13:43:47.971498 9530 catalog_manager.cc:4841] TS 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227): tablet 32cfdb663cd94a2cbe800c879b50cba5 (table test-table [id=fda0f71ea5354d2998ffbc5057573b8a]) successfully deleted
I20260927 13:43:47.992008 10065 raft_consensus.cc:482] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:47.992195 10065 raft_consensus.cc:504] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.992894 10065 leader_election.cc:288] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:47.993355 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1199820b0b43a9b700227c077bf362" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031" is_pre_election: true
I20260927 13:43:47.993446 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1199820b0b43a9b700227c077bf362" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e" is_pre_election: true
I20260927 13:43:47.993578 9701 raft_consensus.cc:2454] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:47.993685 9983 raft_consensus.cc:2454] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:47.994020 9772 leader_election.cc:302] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, cf5c81f81ed54a35bc12a0224401d9f5; no voters:
I20260927 13:43:47.994323 10065 raft_consensus.cc:2781] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:47.994444 10065 raft_consensus.cc:482] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:47.994551 10065 raft_consensus.cc:3037] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.996622 10065 raft_consensus.cc:504] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:47.997205 10065 leader_election.cc:288] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Requested vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:47.997583 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1199820b0b43a9b700227c077bf362" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031"
I20260927 13:43:47.997676 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1199820b0b43a9b700227c077bf362" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e"
I20260927 13:43:47.997776 9701 raft_consensus.cc:3037] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.997865 9983 raft_consensus.cc:3037] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:47.999856 9983 raft_consensus.cc:2454] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:48.000247 9773 leader_election.cc:302] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf5c81f81ed54a35bc12a0224401d9f5, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:48.000485 10065 raft_consensus.cc:2781] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:48.000630 9701 raft_consensus.cc:2454] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:48.000670 10065 raft_consensus.cc:686] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Becoming Leader. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Running, Role: LEADER
I20260927 13:43:48.000914 10065 consensus_queue.cc:237] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.003525 9545 catalog_manager.cc:5495] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } health_report { overall_health: UNKNOWN } } }
Sep 27 13:43:48 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516624, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.254@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:48.055877 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.041387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.1.237.254:43205)
0927 13:43:48.041631 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.041639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.042159 (+ 520us) server_negotiation.cc:407] Connection header received
0927 13:43:48.042311 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.042315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.042368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.042455 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:48.043843 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.044544 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.046386 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.046781 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.046865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.050419 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:48.050424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:48.050426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:48.050452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:48.053159 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:48.053807 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:48.053815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:48.053819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:48.053952 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:48.054327 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:48.054333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:48.054336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:48.054580 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:48.054732 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.055186 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.055460 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":108}
I20260927 13:43:48.055919 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.041431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43205 (local address 127.0.0.1:58056)
0927 13:43:48.042042 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.042072 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.042209 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.042506 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.042512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.043397 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:48.043715 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.043728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.044663 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.044669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.046257 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.046271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.046938 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.046944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.047125 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.047816 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:48.047839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:48.050265 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:48.053300 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:48.053311 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:48.053319 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:48.053696 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:48.054053 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:48.054072 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:48.054076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:48.054214 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:48.054716 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:48.054724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:48.054867 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.055014 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.055345 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":344,"threads_started":1}
I20260927 13:43:48.060842 9545 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58056:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 13:43:48.061792 9545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:48.068307 9680 tablet_service.cc:1467] Processing CreateTablet for tablet 9078a47e5bab44408a7837592ae96931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=205c3095f64c4ccabc30a0996cf160c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:48.068383 9823 tablet_service.cc:1467] Processing CreateTablet for tablet 9078a47e5bab44408a7837592ae96931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=205c3095f64c4ccabc30a0996cf160c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:48.068790 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9078a47e5bab44408a7837592ae96931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.068863 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9078a47e5bab44408a7837592ae96931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.073962 10057 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: Bootstrap starting.
I20260927 13:43:48.074604 9959 tablet_service.cc:1467] Processing CreateTablet for tablet 9078a47e5bab44408a7837592ae96931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=205c3095f64c4ccabc30a0996cf160c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 13:43:48.075140 9959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9078a47e5bab44408a7837592ae96931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.075502 10057 tablet_bootstrap.cc:654] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.077523 10057 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: No bootstrap required, opened a new log
I20260927 13:43:48.077689 10057 ts_tablet_manager.cc:1397] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 13:43:48.078225 10057 raft_consensus.cc:348] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.078421 10057 raft_consensus.cc:374] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.078497 10057 raft_consensus.cc:729] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.078722 10057 consensus_queue.cc:260] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.079324 10057 ts_tablet_manager.cc:1428] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 13:43:48.079114 10062 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.079489 10062 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.081929 10054 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: Bootstrap starting.
I20260927 13:43:48.083205 10054 tablet_bootstrap.cc:654] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.084085 10056 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: Bootstrap starting.
I20260927 13:43:48.085186 10054 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: No bootstrap required, opened a new log
I20260927 13:43:48.085350 10054 ts_tablet_manager.cc:1397] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260927 13:43:48.085780 10056 tablet_bootstrap.cc:654] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.085904 10054 raft_consensus.cc:348] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.086059 10054 raft_consensus.cc:374] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.086102 10054 raft_consensus.cc:729] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.086310 10054 consensus_queue.cc:260] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.086843 10054 ts_tablet_manager.cc:1428] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 13:43:48.086701 10060 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.087033 10060 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.092558 10056 tablet_bootstrap.cc:492] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: No bootstrap required, opened a new log
I20260927 13:43:48.092730 10056 ts_tablet_manager.cc:1397] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 13:43:48.093416 10056 raft_consensus.cc:348] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.093647 10056 raft_consensus.cc:374] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.093727 10056 raft_consensus.cc:729] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.093987 10056 consensus_queue.cc:260] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.094517 10065 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.094671 10056 ts_tablet_manager.cc:1428] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 13:43:48.094810 10065 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.234491 10062 raft_consensus.cc:482] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:48.234711 10062 raft_consensus.cc:504] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.236227 10062 leader_election.cc:288] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:48.243875 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.237049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:45137 (local address 127.1.237.195:44097)
0927 13:43:48.237242 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.237248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.237964 (+ 716us) server_negotiation.cc:407] Connection header received
0927 13:43:48.238198 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.238203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.238281 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.238342 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:48.239058 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.239926 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.241641 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.242778 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.242962 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.243037 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.243464 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.243594 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:43:48.244175 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.236791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:38597 (local address 127.1.237.194:42799)
0927 13:43:48.237025 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.237033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.237655 (+ 622us) server_negotiation.cc:407] Connection header received
0927 13:43:48.237917 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.237923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.238004 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.238076 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:48.238556 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.239473 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.241941 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.243255 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.243426 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.243499 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.243746 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.243868 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:43:48.244542 10094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.236527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:42799 (local address 127.1.237.193:38597)
0927 13:43:48.237545 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.237569 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.237765 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.238100 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.238105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.238128 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:48.238415 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.238423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.239617 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.239623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.241770 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.241785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.243447 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.243453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.243602 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.243623 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.243724 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.243841 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":1,"thread_start_us":1034,"threads_started":1}
I20260927 13:43:48.244843 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.236699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:44097 (local address 127.1.237.193:45137)
0927 13:43:48.237858 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.237883 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.238014 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.238520 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.238524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.238547 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:48.238885 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.238895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.240058 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.240064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.241496 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.241507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.243024 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.243031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.243256 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.243294 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.244292 (+ 998us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.244432 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":535,"threads_started":1}
I20260927 13:43:48.245393 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9078a47e5bab44408a7837592ae96931" candidate_uuid: "06167796f3d74a9a9f45c88b88d78031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e" is_pre_election: true
I20260927 13:43:48.245401 9845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9078a47e5bab44408a7837592ae96931" candidate_uuid: "06167796f3d74a9a9f45c88b88d78031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" is_pre_election: true
I20260927 13:43:48.245658 9983 raft_consensus.cc:2454] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06167796f3d74a9a9f45c88b88d78031 in term 0.
I20260927 13:43:48.245844 9845 raft_consensus.cc:2454] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06167796f3d74a9a9f45c88b88d78031 in term 0.
I20260927 13:43:48.246166 9634 leader_election.cc:302] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:48.246547 10062 raft_consensus.cc:2781] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:48.246672 10062 raft_consensus.cc:482] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:48.246771 10062 raft_consensus.cc:3037] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.248733 10062 raft_consensus.cc:504] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.249302 10062 leader_election.cc:288] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [CANDIDATE]: Term 1 election: Requested vote from peers cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:48.249640 9845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9078a47e5bab44408a7837592ae96931" candidate_uuid: "06167796f3d74a9a9f45c88b88d78031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5c81f81ed54a35bc12a0224401d9f5"
I20260927 13:43:48.249843 9845 raft_consensus.cc:3037] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.249794 9983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9078a47e5bab44408a7837592ae96931" candidate_uuid: "06167796f3d74a9a9f45c88b88d78031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e"
I20260927 13:43:48.250008 9983 raft_consensus.cc:3037] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.252782 9983 raft_consensus.cc:2454] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06167796f3d74a9a9f45c88b88d78031 in term 1.
I20260927 13:43:48.253121 9634 leader_election.cc:302] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:48.253368 9845 raft_consensus.cc:2454] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06167796f3d74a9a9f45c88b88d78031 in term 1.
I20260927 13:43:48.253424 10062 raft_consensus.cc:2781] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:48.253962 10062 raft_consensus.cc:686] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [term 1 LEADER]: Becoming Leader. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Running, Role: LEADER
I20260927 13:43:48.254383 10062 consensus_queue.cc:237] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.257261 10095 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: TxnStatusTablet state changed. Reason: New leader 06167796f3d74a9a9f45c88b88d78031. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.257622 10095 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:48.258229 10097 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 13:43:48.258425 10097 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 13:43:48.258201 9545 catalog_manager.cc:5495] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:48.280640 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.275415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35227 (local address 127.0.0.1:43492)
0927 13:43:48.275654 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.275669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.275761 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.276302 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.276307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.276318 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:48.276502 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.276511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.277552 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.277559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.278509 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.278520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.279181 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.279189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.279378 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.279424 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.280136 (+ 712us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.280162 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.280244 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.280376 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:43:48.280665 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.275420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.1.237.193:35227)
0927 13:43:48.276006 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.276010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.276022 (+ 12us) server_negotiation.cc:407] Connection header received
0927 13:43:48.276078 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.276081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.276139 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.276285 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:48.276620 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.277439 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.278636 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.279015 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.279138 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.279598 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.280020 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.280160 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.280269 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.280340 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260927 13:43:48.284138 9983 raft_consensus.cc:1261] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Refusing update from remote peer 06167796f3d74a9a9f45c88b88d78031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.284551 9845 raft_consensus.cc:1261] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Refusing update from remote peer 06167796f3d74a9a9f45c88b88d78031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.284896 10062 consensus_queue.cc:1035] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.285439 10095 consensus_queue.cc:1035] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.291951 10065 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: TxnStatusTablet state changed. Reason: New leader 06167796f3d74a9a9f45c88b88d78031. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.293287 10065 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.294099 10060 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: TxnStatusTablet state changed. Reason: New leader 06167796f3d74a9a9f45c88b88d78031. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.294355 10060 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.295965 10062 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.296221 10062 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:48.300294 10060 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.300510 10060 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P f1e34b5e248247b098d7926d16d4cd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.301102 10095 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.301337 10095 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P 06167796f3d74a9a9f45c88b88d78031: This TxnStatusTablet replica's current role is: LEADER
I20260927 13:43:48.301425 10070 mvcc.cc:205] Tried to move back new op lower bound from 7333956109445193728 to 7333956109334335488. Current Snapshot: MvccSnapshot[applied={T|T < 7333956109445193728 or (T in {7333956109445193728})}]
I20260927 13:43:48.302430 10069 mvcc.cc:205] Tried to move back new op lower bound from 7333956109445193728 to 7333956109334335488. Current Snapshot: MvccSnapshot[applied={T|T < 7333956109445193728 or (T in {7333956109445193728})}]
I20260927 13:43:48.305121 10068 mvcc.cc:205] Tried to move back new op lower bound from 7333956109445193728 to 7333956109334335488. Current Snapshot: MvccSnapshot[applied={T|T < 7333956109445193728 or (T in {7333956109445193728})}]
I20260927 13:43:48.305569 10065 tablet_replica.cc:439] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06167796f3d74a9a9f45c88b88d78031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } } }
I20260927 13:43:48.305810 10065 tablet_replica.cc:442] T 9078a47e5bab44408a7837592ae96931 P cf5c81f81ed54a35bc12a0224401d9f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 13:43:48.324561 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.315829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:60461 (local address 127.1.237.193:35227)
0927 13:43:48.316023 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.316029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.316694 (+ 665us) server_negotiation.cc:407] Connection header received
0927 13:43:48.316893 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.316900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.316980 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.317080 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:48.317843 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.319022 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.322332 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.323037 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.323202 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.323761 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.323906 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.324013 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.324171 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.324250 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:43:48.324687 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.315896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35227 (local address 127.1.237.194:60461)
0927 13:43:48.316578 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.316597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.316742 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.317112 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.317118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.317131 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:48.317470 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.317481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.320300 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.320337 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.322174 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.322191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.323259 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.323266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.323467 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.323541 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.324026 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.324041 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.324134 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.324279 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":301,"threads_started":1}
I20260927 13:43:48.335768 9701 raft_consensus.cc:1261] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.335789 9983 raft_consensus.cc:1261] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.336326 10065 consensus_queue.cc:1035] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.336616 10061 consensus_queue.cc:1035] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.390506 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.383374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:49343 (local address 127.1.237.194:42799)
0927 13:43:48.383541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.383547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.384234 (+ 687us) server_negotiation.cc:407] Connection header received
0927 13:43:48.384406 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.384413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.384488 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.384585 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:48.385174 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.386313 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.388533 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.389192 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.389324 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.389753 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.389881 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.389956 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.390153 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.390226 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:43:48.391479 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.383461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:42799 (local address 127.1.237.193:49343)
0927 13:43:48.384121 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.384144 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.384278 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.384611 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.384616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.384629 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 13:43:48.385004 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.385014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.386463 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.386468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.388394 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.388407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.389361 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.389369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.389536 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.389594 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.390002 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.390012 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.390918 (+ 906us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.391049 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":329,"threads_started":1}
I20260927 13:43:48.423306 9545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:48.423566 9545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:48.426396 9545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Sending DeleteTablet for 3 replicas of tablet be1199820b0b43a9b700227c077bf362
I20260927 13:43:48.427215 9680 tablet_service.cc:1514] Processing DeleteTablet for tablet be1199820b0b43a9b700227c077bf362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43480
I20260927 13:43:48.427402 9959 tablet_service.cc:1514] Processing DeleteTablet for tablet be1199820b0b43a9b700227c077bf362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:47578
I20260927 13:43:48.427471 10079 tablet_replica.cc:330] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: stopping tablet replica
I20260927 13:43:48.427671 10079 raft_consensus.cc:2229] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:48.427852 10079 raft_consensus.cc:2258] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.427888 9823 tablet_service.cc:1514] Processing DeleteTablet for tablet be1199820b0b43a9b700227c077bf362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:59780
I20260927 13:43:48.428148 10077 tablet_replica.cc:330] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: stopping tablet replica
I20260927 13:43:48.428171 10078 tablet_replica.cc:330] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: stopping tablet replica
I20260927 13:43:48.428347 10078 raft_consensus.cc:2229] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:48.428347 10077 raft_consensus.cc:2229] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:48.428509 10077 raft_consensus.cc:2258] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.428637 10078 raft_consensus.cc:2258] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.429806 10077 ts_tablet_manager.cc:1905] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.436041 10078 ts_tablet_manager.cc:1905] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.442525 10078 ts_tablet_manager.cc:1918] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:48.443580 10079 ts_tablet_manager.cc:1905] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.450371 10079 ts_tablet_manager.cc:1918] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:48.451992 10077 ts_tablet_manager.cc:1918] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:48.452201 10078 log.cc:1196] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/be1199820b0b43a9b700227c077bf362
I20260927 13:43:48.452674 10078 ts_tablet_manager.cc:1939] T be1199820b0b43a9b700227c077bf362 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting consensus metadata
I20260927 13:43:48.452857 10079 log.cc:1196] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/be1199820b0b43a9b700227c077bf362
I20260927 13:43:48.453049 10077 log.cc:1196] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/be1199820b0b43a9b700227c077bf362
I20260927 13:43:48.453510 10079 ts_tablet_manager.cc:1939] T be1199820b0b43a9b700227c077bf362 P 06167796f3d74a9a9f45c88b88d78031: Deleting consensus metadata
I20260927 13:43:48.453975 10077 ts_tablet_manager.cc:1939] T be1199820b0b43a9b700227c077bf362 P f1e34b5e248247b098d7926d16d4cd0e: Deleting consensus metadata
I20260927 13:43:48.455111 9530 catalog_manager.cc:4841] TS cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799): tablet be1199820b0b43a9b700227c077bf362 (table test-table [id=8d64973b60144522924cd05240c8d142]) successfully deleted
I20260927 13:43:48.456275 9531 catalog_manager.cc:4841] TS f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097): tablet be1199820b0b43a9b700227c077bf362 (table test-table [id=8d64973b60144522924cd05240c8d142]) successfully deleted
I20260927 13:43:48.456657 9530 catalog_manager.cc:4841] TS 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227): tablet be1199820b0b43a9b700227c077bf362 (table test-table [id=8d64973b60144522924cd05240c8d142]) successfully deleted
Sep 27 13:43:48 dist-test-slave-7g3l krb5kdc[9500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 13:43:48 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:48 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:48.490898 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:48.480166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.1.237.254:43205)
0927 13:43:48.480338 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.480346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.480817 (+ 471us) server_negotiation.cc:407] Connection header received
0927 13:43:48.480980 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.480986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.481080 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.481180 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:48.482513 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.483662 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.484770 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.485409 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.485529 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.490712 (+ 5183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58060: BlockingRecv error: recv got EOF from 127.0.0.1:58060 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 13:43:48 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM
I20260927 13:43:48.507452 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.494098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.1.237.254:43205)
0927 13:43:48.494305 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.494309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.495697 (+ 1388us) server_negotiation.cc:407] Connection header received
0927 13:43:48.495866 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.495871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.495933 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.496028 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:48.497005 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.497705 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.498814 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.499254 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.499362 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.502661 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:48.502666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:48.502669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:48.502696 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:48.505071 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:48.505651 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:48.505663 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:48.505666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:48.505772 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:48.506129 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:48.506136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:48.506139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:48.506393 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:48.506618 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.506890 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.507093 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:43:48.510520 9545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:48.511122 9545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:48.517431 9680 tablet_service.cc:1467] Processing CreateTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 (DEFAULT_TABLE table=test-table [id=f2e5b7de9958487f96c3f9061079d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.517851 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb511ce163ed4417ba58d208b8d3dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.522622 10057 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Bootstrap starting.
I20260927 13:43:48.523409 9959 tablet_service.cc:1467] Processing CreateTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 (DEFAULT_TABLE table=test-table [id=f2e5b7de9958487f96c3f9061079d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.523814 9959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb511ce163ed4417ba58d208b8d3dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.524327 10057 tablet_bootstrap.cc:654] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.528190 9823 tablet_service.cc:1467] Processing CreateTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 (DEFAULT_TABLE table=test-table [id=f2e5b7de9958487f96c3f9061079d4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.528702 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb511ce163ed4417ba58d208b8d3dfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.534309 10054 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Bootstrap starting.
I20260927 13:43:48.535627 10057 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: No bootstrap required, opened a new log
I20260927 13:43:48.535748 10054 tablet_bootstrap.cc:654] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.535815 10057 ts_tablet_manager.cc:1397] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260927 13:43:48.536546 10057 raft_consensus.cc:348] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.536839 10057 raft_consensus.cc:374] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.536955 10057 raft_consensus.cc:729] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.537045 10056 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Bootstrap starting.
I20260927 13:43:48.537181 10057 consensus_queue.cc:260] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.538221 10057 ts_tablet_manager.cc:1428] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:48.538771 10056 tablet_bootstrap.cc:654] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.540030 10054 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: No bootstrap required, opened a new log
I20260927 13:43:48.540153 10054 ts_tablet_manager.cc:1397] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 13:43:48.541173 10054 raft_consensus.cc:348] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.541397 10054 raft_consensus.cc:374] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.541486 10054 raft_consensus.cc:729] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.541695 10054 consensus_queue.cc:260] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.542167 10054 ts_tablet_manager.cc:1428] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:48.544358 10056 tablet_bootstrap.cc:492] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: No bootstrap required, opened a new log
I20260927 13:43:48.544505 10056 ts_tablet_manager.cc:1397] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260927 13:43:48.545764 10056 raft_consensus.cc:348] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.545931 10056 raft_consensus.cc:374] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.546032 10056 raft_consensus.cc:729] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.546250 10056 consensus_queue.cc:260] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.546691 10056 ts_tablet_manager.cc:1428] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 13:43:48.716307 10060 raft_consensus.cc:482] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:48.716511 10060 raft_consensus.cc:504] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.717890 10060 leader_election.cc:288] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799)
I20260927 13:43:48.724649 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.718193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:52211 (local address 127.1.237.193:35227)
0927 13:43:48.718385 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.718392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.718987 (+ 595us) server_negotiation.cc:407] Connection header received
0927 13:43:48.719085 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.719090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.719184 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.719305 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:48.719861 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.720787 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.722691 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.723839 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.724004 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.724086 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.724307 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.724431 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:43:48.724786 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.718157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35227 (local address 127.1.237.195:52211)
0927 13:43:48.718641 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.718662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.718809 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.719322 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.719327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.719349 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:48.719718 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.719730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.720911 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.720916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.722535 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.722546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.724024 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.724028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.724145 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.724167 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.724256 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.724385 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":46,"thread_start_us":212,"threads_started":1}
I20260927 13:43:48.725181 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.718878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:37403 (local address 127.1.237.194:42799)
0927 13:43:48.719117 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.719123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.719399 (+ 276us) server_negotiation.cc:407] Connection header received
0927 13:43:48.719653 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.719658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.719731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.719827 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:43:48.720429 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.721202 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.722998 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.724200 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.724386 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.724444 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.724834 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.724971 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:43:48.725471 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb511ce163ed4417ba58d208b8d3dfa9" candidate_uuid: "f1e34b5e248247b098d7926d16d4cd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031" is_pre_election: true
I20260927 13:43:48.725696 9701 raft_consensus.cc:2454] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e34b5e248247b098d7926d16d4cd0e in term 0.
I20260927 13:43:48.725837 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.718675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:42799 (local address 127.1.237.195:37403)
0927 13:43:48.719306 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:48.719318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:48.719462 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:48.719851 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:48.719855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:48.719885 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:43:48.720277 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.720289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.721320 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.721323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.722842 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:48.722853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.724448 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.724453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:48.724629 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.724665 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:43:48.725378 (+ 713us) client_negotiation.cc:240] Negotiation successful
0927 13:43:48.725482 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
I20260927 13:43:48.726262 9911 leader_election.cc:302] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:48.726639 10060 raft_consensus.cc:2781] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:48.726759 10060 raft_consensus.cc:482] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:48.726861 10060 raft_consensus.cc:3037] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.726979 9845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb511ce163ed4417ba58d208b8d3dfa9" candidate_uuid: "f1e34b5e248247b098d7926d16d4cd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" is_pre_election: true
I20260927 13:43:48.727254 9845 raft_consensus.cc:2454] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e34b5e248247b098d7926d16d4cd0e in term 0.
I20260927 13:43:48.729094 10060 raft_consensus.cc:504] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.729583 10060 leader_election.cc:288] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 election: Requested vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799)
I20260927 13:43:48.730010 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb511ce163ed4417ba58d208b8d3dfa9" candidate_uuid: "f1e34b5e248247b098d7926d16d4cd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031"
I20260927 13:43:48.730098 9845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb511ce163ed4417ba58d208b8d3dfa9" candidate_uuid: "f1e34b5e248247b098d7926d16d4cd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5c81f81ed54a35bc12a0224401d9f5"
I20260927 13:43:48.730204 9701 raft_consensus.cc:3037] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.730304 9845 raft_consensus.cc:3037] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:48.732272 9701 raft_consensus.cc:2454] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e34b5e248247b098d7926d16d4cd0e in term 1.
I20260927 13:43:48.732355 9845 raft_consensus.cc:2454] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e34b5e248247b098d7926d16d4cd0e in term 1.
I20260927 13:43:48.732621 9911 leader_election.cc:302] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:48.732872 10060 raft_consensus.cc:2781] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:48.733201 10060 raft_consensus.cc:686] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 1 LEADER]: Becoming Leader. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Running, Role: LEADER
I20260927 13:43:48.733559 10060 consensus_queue.cc:237] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.736479 9545 catalog_manager.cc:5495] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195). New cstate: current_term: 1 leader_uuid: "f1e34b5e248247b098d7926d16d4cd0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260927 13:43:48.781769 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:48.777011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.1.237.195:44097)
0927 13:43:48.777386 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:48.777390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:48.777405 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:43:48.777546 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:48.777549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:48.777629 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:48.777757 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:48.778074 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.778750 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.780005 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:48.780398 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:48.780483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:48.780808 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:48.781299 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:48.781403 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:48.781501 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 13:43:48.781554 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":221,"threads_started":1}
I20260927 13:43:48.784960 9701 raft_consensus.cc:1261] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Refusing update from remote peer f1e34b5e248247b098d7926d16d4cd0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.784989 9845 raft_consensus.cc:1261] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Refusing update from remote peer f1e34b5e248247b098d7926d16d4cd0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:48.785585 10129 consensus_queue.cc:1035] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.785912 10060 consensus_queue.cc:1035] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:48.804167 9545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:48.804379 9545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:48.807051 9545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Sending DeleteTablet for 3 replicas of tablet bb511ce163ed4417ba58d208b8d3dfa9
I20260927 13:43:48.807821 9680 tablet_service.cc:1514] Processing DeleteTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43480
I20260927 13:43:48.807861 9823 tablet_service.cc:1514] Processing DeleteTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:59780
I20260927 13:43:48.808136 10078 tablet_replica.cc:330] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: stopping tablet replica
I20260927 13:43:48.808102 9959 tablet_service.cc:1514] Processing DeleteTablet for tablet bb511ce163ed4417ba58d208b8d3dfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:48 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:47578
I20260927 13:43:48.808307 10078 raft_consensus.cc:2229] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:48.808357 10079 tablet_replica.cc:330] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: stopping tablet replica
I20260927 13:43:48.808382 10077 tablet_replica.cc:330] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: stopping tablet replica
I20260927 13:43:48.808517 10079 raft_consensus.cc:2229] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:48.808526 10078 raft_consensus.cc:2258] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.808549 10077 raft_consensus.cc:2229] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:48.808712 10079 raft_consensus.cc:2258] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.809010 10077 raft_consensus.cc:2258] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:48.809638 10079 ts_tablet_manager.cc:1905] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.810001 10078 ts_tablet_manager.cc:1905] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.810192 9545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:43:48.810748 9545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:43:48.811939 10077 ts_tablet_manager.cc:1905] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:48.815958 10079 ts_tablet_manager.cc:1918] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:48.816108 10079 log.cc:1196] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/bb511ce163ed4417ba58d208b8d3dfa9
I20260927 13:43:48.816577 10079 ts_tablet_manager.cc:1939] T bb511ce163ed4417ba58d208b8d3dfa9 P 06167796f3d74a9a9f45c88b88d78031: Deleting consensus metadata
I20260927 13:43:48.817916 9530 catalog_manager.cc:4841] TS 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227): tablet bb511ce163ed4417ba58d208b8d3dfa9 (table test-table [id=f2e5b7de9958487f96c3f9061079d4ba]) successfully deleted
I20260927 13:43:48.818308 9680 tablet_service.cc:1467] Processing CreateTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 (DEFAULT_TABLE table=test-table [id=da2e85788502414a8fb1b0c063f8cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.818660 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ddcbd62c7f44838cdeb77ca5361891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.819808 10078 ts_tablet_manager.cc:1918] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:48.819908 10078 log.cc:1196] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/bb511ce163ed4417ba58d208b8d3dfa9
I20260927 13:43:48.820185 10078 ts_tablet_manager.cc:1939] T bb511ce163ed4417ba58d208b8d3dfa9 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting consensus metadata
I20260927 13:43:48.821053 9959 tablet_service.cc:1467] Processing CreateTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 (DEFAULT_TABLE table=test-table [id=da2e85788502414a8fb1b0c063f8cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.821431 9959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ddcbd62c7f44838cdeb77ca5361891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.823273 9530 catalog_manager.cc:4841] TS cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799): tablet bb511ce163ed4417ba58d208b8d3dfa9 (table test-table [id=f2e5b7de9958487f96c3f9061079d4ba]) successfully deleted
I20260927 13:43:48.823314 10077 ts_tablet_manager.cc:1918] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:43:48.823467 10077 log.cc:1196] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/bb511ce163ed4417ba58d208b8d3dfa9
I20260927 13:43:48.823519 9823 tablet_service.cc:1467] Processing CreateTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 (DEFAULT_TABLE table=test-table [id=da2e85788502414a8fb1b0c063f8cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:43:48.823858 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ddcbd62c7f44838cdeb77ca5361891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:48.823910 10077 ts_tablet_manager.cc:1939] T bb511ce163ed4417ba58d208b8d3dfa9 P f1e34b5e248247b098d7926d16d4cd0e: Deleting consensus metadata
I20260927 13:43:48.827673 9531 catalog_manager.cc:4841] TS f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097): tablet bb511ce163ed4417ba58d208b8d3dfa9 (table test-table [id=f2e5b7de9958487f96c3f9061079d4ba]) successfully deleted
I20260927 13:43:48.828004 10054 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Bootstrap starting.
I20260927 13:43:48.829207 10054 tablet_bootstrap.cc:654] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.831643 10057 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Bootstrap starting.
I20260927 13:43:48.832177 10054 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: No bootstrap required, opened a new log
I20260927 13:43:48.832350 10054 ts_tablet_manager.cc:1397] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 13:43:48.833169 10057 tablet_bootstrap.cc:654] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:48.833122 10054 raft_consensus.cc:348] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.833391 10054 raft_consensus.cc:374] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.833479 10054 raft_consensus.cc:729] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e34b5e248247b098d7926d16d4cd0e, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.833711 10054 consensus_queue.cc:260] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.833966 10129 raft_consensus.cc:482] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:48.834291 10054 ts_tablet_manager.cc:1428] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:48.834156 10129 raft_consensus.cc:504] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.834764 10129 leader_election.cc:288] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799)
I20260927 13:43:48.836074 10056 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Bootstrap starting.
W20260927 13:43:48.836180 9911 leader_election.cc:341] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799): Not found: Tablet not found: d1ddcbd62c7f44838cdeb77ca5361891
I20260927 13:43:48.837224 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ddcbd62c7f44838cdeb77ca5361891" candidate_uuid: "f1e34b5e248247b098d7926d16d4cd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031" is_pre_election: true
I20260927 13:43:48.837344 10056 tablet_bootstrap.cc:654] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Neither blocks nor log segments found. Creating new log.
W20260927 13:43:48.837800 9911 leader_election.cc:341] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227): Illegal state: must be running to vote when last-logged opid is not known
I20260927 13:43:48.837931 9911 leader_election.cc:302] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e34b5e248247b098d7926d16d4cd0e; no voters: 06167796f3d74a9a9f45c88b88d78031, cf5c81f81ed54a35bc12a0224401d9f5
I20260927 13:43:48.838680 10135 raft_consensus.cc:2726] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:43:48.844612 10056 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: No bootstrap required, opened a new log
I20260927 13:43:48.844758 10056 ts_tablet_manager.cc:1397] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260927 13:43:48.845301 10056 raft_consensus.cc:348] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.845477 10056 raft_consensus.cc:374] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.845568 10056 raft_consensus.cc:729] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.845810 10056 consensus_queue.cc:260] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.846235 10056 ts_tablet_manager.cc:1428] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 13:43:48.847417 10057 tablet_bootstrap.cc:492] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: No bootstrap required, opened a new log
I20260927 13:43:48.847625 10057 ts_tablet_manager.cc:1397] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260927 13:43:48.848399 10057 raft_consensus.cc:348] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.848704 10057 raft_consensus.cc:374] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:48.848788 10057 raft_consensus.cc:729] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06167796f3d74a9a9f45c88b88d78031, State: Initialized, Role: FOLLOWER
I20260927 13:43:48.849009 10057 consensus_queue.cc:260] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:48.849496 10057 ts_tablet_manager.cc:1428] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 13:43:49.050017 10061 raft_consensus.cc:482] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:43:49.050364 10061 raft_consensus.cc:504] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:49.051002 10061 leader_election.cc:288] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:49.051673 9982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ddcbd62c7f44838cdeb77ca5361891" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e" is_pre_election: true
I20260927 13:43:49.051672 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ddcbd62c7f44838cdeb77ca5361891" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031" is_pre_election: true
I20260927 13:43:49.051985 9982 raft_consensus.cc:2454] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:49.051973 9701 raft_consensus.cc:2454] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 0.
I20260927 13:43:49.052423 9772 leader_election.cc:302] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06167796f3d74a9a9f45c88b88d78031, cf5c81f81ed54a35bc12a0224401d9f5; no voters:
I20260927 13:43:49.052736 10061 raft_consensus.cc:2781] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:43:49.052850 10061 raft_consensus.cc:482] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:43:49.052970 10061 raft_consensus.cc:3037] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:49.056052 10061 raft_consensus.cc:504] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:49.056727 10061 leader_election.cc:288] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Requested vote from peers 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227), f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097)
I20260927 13:43:49.057250 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ddcbd62c7f44838cdeb77ca5361891" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06167796f3d74a9a9f45c88b88d78031"
I20260927 13:43:49.057278 9982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ddcbd62c7f44838cdeb77ca5361891" candidate_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e34b5e248247b098d7926d16d4cd0e"
I20260927 13:43:49.057487 9701 raft_consensus.cc:3037] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:49.057492 9982 raft_consensus.cc:3037] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:49.059648 9982 raft_consensus.cc:2454] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:49.060031 9773 leader_election.cc:302] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf5c81f81ed54a35bc12a0224401d9f5, f1e34b5e248247b098d7926d16d4cd0e; no voters:
I20260927 13:43:49.060256 9701 raft_consensus.cc:2454] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5c81f81ed54a35bc12a0224401d9f5 in term 1.
I20260927 13:43:49.060287 10061 raft_consensus.cc:2781] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:49.061085 10061 raft_consensus.cc:686] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Becoming Leader. State: Replica: cf5c81f81ed54a35bc12a0224401d9f5, State: Running, Role: LEADER
I20260927 13:43:49.061543 10061 consensus_queue.cc:237] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } }
I20260927 13:43:49.064204 9543 catalog_manager.cc:5495] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5c81f81ed54a35bc12a0224401d9f5" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20260927 13:43:49.090597 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:49.085355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.1.237.194:42799)
0927 13:43:49.085545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:49.085549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:49.085855 (+ 306us) server_negotiation.cc:407] Connection header received
0927 13:43:49.086048 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:49.086051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:49.086107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:49.086175 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:43:49.086636 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.087348 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.089142 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.089490 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.089572 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:49.090046 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:43:49.090143 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:43:49.090203 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:49.090363 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 13:43:49.090425 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 13:43:49.098737 9982 raft_consensus.cc:1261] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:49.098737 9701 raft_consensus.cc:1261] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Refusing update from remote peer cf5c81f81ed54a35bc12a0224401d9f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:43:49.099304 10140 consensus_queue.cc:1035] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e34b5e248247b098d7926d16d4cd0e" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:49.099641 10061 consensus_queue.cc:1035] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06167796f3d74a9a9f45c88b88d78031" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:43:49.151523 9543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:49.151753 9543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:43:49.154299 9543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ca24c07817643329748af6165849fe3: Sending DeleteTablet for 3 replicas of tablet d1ddcbd62c7f44838cdeb77ca5361891
I20260927 13:43:49.155081 9680 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:49 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43480
I20260927 13:43:49.155102 9959 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:49 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:47578
I20260927 13:43:49.155303 9823 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ddcbd62c7f44838cdeb77ca5361891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:43:49 UTC) from {username='oryx', principal='oryx/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:59780
I20260927 13:43:49.155510 10079 tablet_replica.cc:330] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: stopping tablet replica
I20260927 13:43:49.155611 10077 tablet_replica.cc:330] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: stopping tablet replica
I20260927 13:43:49.155609 10078 tablet_replica.cc:330] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: stopping tablet replica
I20260927 13:43:49.155715 10079 raft_consensus.cc:2229] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:49.155799 10077 raft_consensus.cc:2229] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:43:49.155802 10078 raft_consensus.cc:2229] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:43:49.155901 10079 raft_consensus.cc:2258] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:49.155988 10077 raft_consensus.cc:2258] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:49.156104 10078 raft_consensus.cc:2258] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:43:49.157059 10079 ts_tablet_manager.cc:1905] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:49.157124 10077 ts_tablet_manager.cc:1905] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:49.157138 10078 ts_tablet_manager.cc:1905] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:43:49.160254 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9616
I20260927 13:43:49.163801 10079 ts_tablet_manager.cc:1918] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:49.163918 10078 ts_tablet_manager.cc:1918] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:49.164017 10079 log.cc:1196] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/d1ddcbd62c7f44838cdeb77ca5361891
I20260927 13:43:49.164179 10078 log.cc:1196] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/d1ddcbd62c7f44838cdeb77ca5361891
I20260927 13:43:49.164453 10079 ts_tablet_manager.cc:1939] T d1ddcbd62c7f44838cdeb77ca5361891 P 06167796f3d74a9a9f45c88b88d78031: Deleting consensus metadata
I20260927 13:43:49.164674 10078 ts_tablet_manager.cc:1939] T d1ddcbd62c7f44838cdeb77ca5361891 P cf5c81f81ed54a35bc12a0224401d9f5: Deleting consensus metadata
I20260927 13:43:49.165577 9530 catalog_manager.cc:4841] TS 06167796f3d74a9a9f45c88b88d78031 (127.1.237.193:35227): tablet d1ddcbd62c7f44838cdeb77ca5361891 (table test-table [id=da2e85788502414a8fb1b0c063f8cbfd]) successfully deleted
I20260927 13:43:49.165951 9530 catalog_manager.cc:4841] TS cf5c81f81ed54a35bc12a0224401d9f5 (127.1.237.194:42799): tablet d1ddcbd62c7f44838cdeb77ca5361891 (table test-table [id=da2e85788502414a8fb1b0c063f8cbfd]) successfully deleted
I20260927 13:43:49.168469 10077 ts_tablet_manager.cc:1918] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 13:43:49.168634 10077 log.cc:1196] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/d1ddcbd62c7f44838cdeb77ca5361891
I20260927 13:43:49.169111 10077 ts_tablet_manager.cc:1939] T d1ddcbd62c7f44838cdeb77ca5361891 P f1e34b5e248247b098d7926d16d4cd0e: Deleting consensus metadata
I20260927 13:43:49.170089 9531 catalog_manager.cc:4841] TS f1e34b5e248247b098d7926d16d4cd0e (127.1.237.195:44097): tablet d1ddcbd62c7f44838cdeb77ca5361891 (table test-table [id=da2e85788502414a8fb1b0c063f8cbfd]) successfully deleted
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.193@KRBTEST.COM
I20260927 13:43:49.181589 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:49.162365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.1.237.193:35227)
0927 13:43:49.162530 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:49.162536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:49.165753 (+ 3217us) server_negotiation.cc:407] Connection header received
0927 13:43:49.166012 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:49.166018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:49.166101 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:49.166165 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:49.167785 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.168844 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.170484 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.171073 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.171237 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:49.175193 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:49.175213 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:49.175232 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:49.175281 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:49.178957 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.179606 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.179615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.179619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.179767 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.180112 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.180118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.180124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.180537 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:49.180830 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:49.181017 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 13:43:49.181220 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:43:49.182271 9746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:49.297328 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9616
I20260927 13:43:49.319283 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9755
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.194@KRBTEST.COM
I20260927 13:43:49.334051 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:49.319968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.1.237.194:42799)
0927 13:43:49.320133 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:49.320136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:49.320184 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:43:49.320334 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:49.320338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:49.320395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:49.320483 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:49.321961 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.323065 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.324530 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.324973 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.325097 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:49.328841 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:49.328857 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:49.328879 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:49.328907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:49.331473 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.332133 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.332142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.332145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.332266 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.332619 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.332624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.332627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.332964 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:49.333271 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:49.333529 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 13:43:49.333733 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:43:49.334654 9885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:49.459965 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9755
I20260927 13:43:49.481958 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9894
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[9500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.237.195@KRBTEST.COM
I20260927 13:43:49.495122 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:49.482627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.1.237.195:44097)
0927 13:43:49.483060 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:49.483064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:49.483079 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:43:49.483127 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:49.483131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:49.483254 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:49.483349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:49.484563 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.485239 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.486317 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:49.486658 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:49.486748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:49.490601 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:49.490614 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:49.490630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:49.490656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:49.493106 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.493633 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.493639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.493641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.493780 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:49.494064 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:49.494067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:49.494069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:49.494376 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:49.494590 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:49.494765 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 13:43:49.494904 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":240,"threads_started":1}
I20260927 13:43:49.495697 10024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:49.611899 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9894
I20260927 13:43:49.632340 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 9513
I20260927 13:43:49.633460 9585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:49.742646 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 9513
2026-09-27T13:43:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6468 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[10158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[10158](info): set up 2 sockets
Sep 27 13:43:49 dist-test-slave-7g3l krb5kdc[10158](info): commencing operation
krb5kdc: starting...
W20260927 13:43:50.800052 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:50 dist-test-slave-7g3l krb5kdc[10158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:50.985971 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42309
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:51.285921 10171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:51.286176 10171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:51.286341 10171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:51.293013 10171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:51.293095 10171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:51.293190 10171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:51.293282 10171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:51.293368 10171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:51.302064 10171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:51.303124 10171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:51.304307 10171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:51.315065 10179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:51.315208 10176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:51.315132 10177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:51.316067 10171 server_base.cc:1034] running on GCE node
I20260927 13:43:51.316735 10171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:51.317898 10171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:51.319087 10171 hybrid_clock.cc:648] HybridClock initialized: now 1790516631319054 us; error 23 us; skew 500 ppm
I20260927 13:43:51.319402 10171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T13:43:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1646 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:43:51 dist-test-slave-7g3l krb5kdc[10185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:43:51 dist-test-slave-7g3l krb5kdc[10185](info): set up 2 sockets
Sep 27 13:43:51 dist-test-slave-7g3l krb5kdc[10185](info): commencing operation
krb5kdc: starting...
W20260927 13:43:52.451997 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:43:52 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:43:52.650429 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42687
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:52.946054 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:52.946355 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:52.946523 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:52.953227 10198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:52.953347 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:52.953441 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:52.953562 10198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:52.953642 10198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:52.962634 10198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:52.963814 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:52.965014 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:52.976032 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:52.976130 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:52.976105 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:52.977010 10198 server_base.cc:1034] running on GCE node
I20260927 13:43:52.977739 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:52.979014 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:52.980250 10198 hybrid_clock.cc:648] HybridClock initialized: now 1790516632980206 us; error 69 us; skew 500 ppm
I20260927 13:43:52.980576 10198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:52 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:52.987573 10198 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:52.988859 10198 webserver.cc:460] Webserver started at http://127.1.237.254:39121/ using document root <none> and password file <none>
I20260927 13:43:52.989481 10198 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:52.989571 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:52.989811 10198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:52.992630 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "9c234e9340c24948833f5febb7613f95"
format_stamp: "Formatted at 2026-09-27 13:43:52 on dist-test-slave-7g3l"
server_key: "a9d551012e1935f73a4d0bee0c76b783"
server_key_iv: "8089f086319be2f84a4facc5e4399135"
server_key_version: "encryptionkey@0"
I20260927 13:43:52.993317 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "9c234e9340c24948833f5febb7613f95"
format_stamp: "Formatted at 2026-09-27 13:43:52 on dist-test-slave-7g3l"
server_key: "a9d551012e1935f73a4d0bee0c76b783"
server_key_iv: "8089f086319be2f84a4facc5e4399135"
server_key_version: "encryptionkey@0"
I20260927 13:43:52.998086 10198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:43:53.001296 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:53.002571 10198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:43:53.002705 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "9c234e9340c24948833f5febb7613f95"
format_stamp: "Formatted at 2026-09-27 13:43:52 on dist-test-slave-7g3l"
server_key: "a9d551012e1935f73a4d0bee0c76b783"
server_key_iv: "8089f086319be2f84a4facc5e4399135"
server_key_version: "encryptionkey@0"
I20260927 13:43:53.002851 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:53.019838 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:53.023411 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:53.023646 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:53.040961 10198 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:42687
I20260927 13:43:53.040995 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:42687 every 8 connection(s)
I20260927 13:43:53.042136 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:53.045529 10265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:43:53.046687 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10198
I20260927 13:43:53.046900 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:43:53.047868 1975 external_mini_cluster.cc:1428] Setting key 83ff7b2b04331fdd106721c4265c9da9
I20260927 13:43:53.059434 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Bootstrap starting.
Sep 27 13:43:53 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:53.062702 10265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Neither blocks nor log segments found. Creating new log.
I20260927 13:43:53.063788 10265 log.cc:824] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:53.066444 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: No bootstrap required, opened a new log
I20260927 13:43:53.069141 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:53.049676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.1.237.254:42687)
0927 13:43:53.050453 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:53.050463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:53.050503 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:43:53.052079 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:53.052098 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:53.052447 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:53.052742 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:53.054375 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.055636 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.057376 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.057957 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.058084 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:53.062414 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:53.062421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:53.062435 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:53.062466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:53.065476 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:53.066093 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:53.066107 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:53.066112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:53.066262 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:53.066641 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:53.066647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:53.066650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:53.067110 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:53.067422 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:53.067818 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 13:43:53.068241 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":388,"threads_started":1}
I20260927 13:43:53.073889 10265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:53.074260 10265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:43:53.074360 10265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c234e9340c24948833f5febb7613f95, State: Initialized, Role: FOLLOWER
I20260927 13:43:53.074914 10265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:53.075134 10265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:53.075369 10265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:53.075547 10265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:43:53.077675 10265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:53.078128 10265 leader_election.cc:302] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c234e9340c24948833f5febb7613f95; no voters:
I20260927 13:43:53.078660 10265 leader_election.cc:288] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:43:53.078781 10270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:43:53.079499 10270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 LEADER]: Becoming Leader. State: Replica: 9c234e9340c24948833f5febb7613f95, State: Running, Role: LEADER
I20260927 13:43:53.080004 10270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:53.080623 10265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:53.082527 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c234e9340c24948833f5febb7613f95. Latest consensus state: current_term: 1 leader_uuid: "9c234e9340c24948833f5febb7613f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } } }
I20260927 13:43:53.082949 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:53.083361 10271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c234e9340c24948833f5febb7613f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } } }
I20260927 13:43:53.083664 10271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:53.083748 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:53.089962 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:53.098165 10279 catalog_manager.cc:1347] Generated new cluster ID: fc089d9499654160ba2802b1a870bb24
I20260927 13:43:53.098317 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:53.113201 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:43:53.114456 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:53.121608 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Generated new TSK 0
I20260927 13:43:53.122318 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:43:53.175916 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:53.507846 10291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:53.508093 10291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:53.508255 10291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:53.515126 10291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:53.515223 10291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:53.515375 10291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:53.524156 10291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:53.525239 10291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:53.526448 10291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:53.538554 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:53.538614 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:53.538647 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:53.539438 10291 server_base.cc:1034] running on GCE node
I20260927 13:43:53.540031 10291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:53.540880 10291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:53.542066 10291 hybrid_clock.cc:648] HybridClock initialized: now 1790516633542022 us; error 37 us; skew 500 ppm
I20260927 13:43:53.542352 10291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:53 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:53.545274 10291 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:43:53.546594 10291 webserver.cc:460] Webserver started at http://127.1.237.193:35977/ using document root <none> and password file <none>
I20260927 13:43:53.547278 10291 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:53.547410 10291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:53.547713 10291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:53.550345 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "6a4f2616f90a4dc98e64e97427d7a84e"
format_stamp: "Formatted at 2026-09-27 13:43:53 on dist-test-slave-7g3l"
server_key: "171f2f780984ef0200cf282322b8f40e"
server_key_iv: "935e352bbf3709202363c1f78f48185e"
server_key_version: "encryptionkey@0"
I20260927 13:43:53.551062 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "6a4f2616f90a4dc98e64e97427d7a84e"
format_stamp: "Formatted at 2026-09-27 13:43:53 on dist-test-slave-7g3l"
server_key: "171f2f780984ef0200cf282322b8f40e"
server_key_iv: "935e352bbf3709202363c1f78f48185e"
server_key_version: "encryptionkey@0"
I20260927 13:43:53.555677 10291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:43:53.558713 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:53.559973 10291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:53.560132 10291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "6a4f2616f90a4dc98e64e97427d7a84e"
format_stamp: "Formatted at 2026-09-27 13:43:53 on dist-test-slave-7g3l"
server_key: "171f2f780984ef0200cf282322b8f40e"
server_key_iv: "935e352bbf3709202363c1f78f48185e"
server_key_version: "encryptionkey@0"
I20260927 13:43:53.560314 10291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:53.592249 10291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:53.595691 10291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:53.595940 10291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:53.596731 10291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:53.597847 10291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:53.597916 10291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:53.598073 10291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:53.598127 10291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:53.628402 10291 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:35299
I20260927 13:43:53.628497 10418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:35299 every 8 connection(s)
I20260927 13:43:53.629567 10291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:53.635933 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10291
I20260927 13:43:53.636119 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:43:53.636891 1975 external_mini_cluster.cc:1428] Setting key 3d35055223aec5282ae502090892de24
Sep 27 13:43:53 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:53.649581 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:53.631985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:34863 (local address 127.1.237.254:42687)
0927 13:43:53.632408 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:53.632414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:53.633282 (+ 868us) server_negotiation.cc:407] Connection header received
0927 13:43:53.634989 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:53.634993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:53.635079 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:53.635229 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:53.637149 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.637976 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.639697 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.640413 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.640547 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:53.643843 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:53.643851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:53.643859 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:53.643906 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:53.646668 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:53.647544 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:53.647547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:53.647549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:53.647652 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:53.648107 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:53.648110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:53.648112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:53.648320 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:53.648415 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:53.649201 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 13:43:53.649345 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":194,"threads_started":1}
I20260927 13:43:53.650982 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:53.632489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:34863)
0927 13:43:53.633142 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:53.633177 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:53.634752 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:53.635480 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:53.635492 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:53.636211 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:53.636962 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:53.636988 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.638116 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.638121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:53.639569 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:53.639580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:53.640608 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:53.640612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:53.640791 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:53.641411 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:53.641434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:53.643640 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:53.646804 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:53.646815 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:53.646860 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:53.647445 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:53.647749 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:53.647756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:53.647763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:53.648007 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:53.648438 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:53.648449 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:53.648825 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 13:43:53.649117 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 13:43:53.649615 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":250,"threads_started":1}
I20260927 13:43:53.652376 10419 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42687
I20260927 13:43:53.652640 10419 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:53.653258 10419 heartbeater.cc:511] Master 127.1.237.254:42687 requested a full tablet report, sending...
I20260927 13:43:53.655055 10230 ts_manager.cc:194] Registered new tserver with Master: 6a4f2616f90a4dc98e64e97427d7a84e (127.1.237.193:35299)
I20260927 13:43:53.656431 10230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:34863
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:43:53.682269 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:53.981309 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:53.981597 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:53.981738 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:53.988911 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:53.988998 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:53.989164 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:53.998395 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:53.999524 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:54.000671 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:54.012888 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:54.012928 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:54.013468 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:54.013724 10425 server_base.cc:1034] running on GCE node
I20260927 13:43:54.014310 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:54.015112 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:54.016358 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790516634016306 us; error 36 us; skew 500 ppm
I20260927 13:43:54.016647 10425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:54.019598 10425 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:43:54.020951 10425 webserver.cc:460] Webserver started at http://127.1.237.194:41343/ using document root <none> and password file <none>
I20260927 13:43:54.021603 10425 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:54.021724 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:54.022037 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:54.024716 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "87e13b3e1e794a32b16f6c8904e1cddd"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "12a5e9518c60822afbe0dbc21e7633f5"
server_key_iv: "4fc96a6877786e7299846f00cb7a9f08"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.025489 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "87e13b3e1e794a32b16f6c8904e1cddd"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "12a5e9518c60822afbe0dbc21e7633f5"
server_key_iv: "4fc96a6877786e7299846f00cb7a9f08"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.030158 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 13:43:54.033384 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.034677 10425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:43:54.034848 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "87e13b3e1e794a32b16f6c8904e1cddd"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "12a5e9518c60822afbe0dbc21e7633f5"
server_key_iv: "4fc96a6877786e7299846f00cb7a9f08"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.035036 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:54.048599 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:54.053144 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:54.053424 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:54.054405 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:54.055857 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:54.055948 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.056054 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:54.056108 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.088644 10425 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:44659
I20260927 13:43:54.088685 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:44659 every 8 connection(s)
I20260927 13:43:54.089780 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:54.098603 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10425
I20260927 13:43:54.098775 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:43:54.099265 1975 external_mini_cluster.cc:1428] Setting key 388fc37ba64aa800d1caf1e8345c19df
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:54.108718 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.092033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:39567 (local address 127.1.237.254:42687)
0927 13:43:54.092199 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:54.092204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:54.093247 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 13:43:54.095017 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:54.095021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:54.095110 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:54.095254 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:54.096540 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.097209 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.098882 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.099411 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.099503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.103084 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:54.103089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:54.103092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:54.103120 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:54.105702 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.106434 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.106441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.106445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.106571 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.106995 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.107002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.107007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.107338 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:54.107491 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:54.108155 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 13:43:54.108382 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:43:54.109660 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.092467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:39567)
0927 13:43:54.093085 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:54.093135 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:54.094793 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:54.095417 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:54.095425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:54.095890 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:54.096407 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:54.096423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.097325 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.097328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:54.098652 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:54.098673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.099607 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.099614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:54.099861 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.100513 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:54.100534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:54.102878 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:54.105849 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:54.105858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:54.105873 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:54.106325 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:54.106695 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:54.106699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:54.106703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:54.106887 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:54.107468 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:54.107476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:54.107801 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 13:43:54.108076 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 13:43:54.108586 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
I20260927 13:43:54.111119 10553 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42687
I20260927 13:43:54.111459 10553 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:54.112259 10553 heartbeater.cc:511] Master 127.1.237.254:42687 requested a full tablet report, sending...
I20260927 13:43:54.113830 10230 ts_manager.cc:194] Registered new tserver with Master: 87e13b3e1e794a32b16f6c8904e1cddd (127.1.237.194:44659)
I20260927 13:43:54.115042 10230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:39567
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:43:54.147138 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:54.449534 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:54.449749 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:54.449922 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:54.456765 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:54.456851 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:54.457052 10558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:54.466006 10558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:54.467070 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:54.468291 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:54.480489 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:54.480454 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:54.480893 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:54.481288 10558 server_base.cc:1034] running on GCE node
I20260927 13:43:54.481799 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:54.482607 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:54.483803 10558 hybrid_clock.cc:648] HybridClock initialized: now 1790516634483738 us; error 42 us; skew 500 ppm
I20260927 13:43:54.484122 10558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:54.486907 10558 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:43:54.488287 10558 webserver.cc:460] Webserver started at http://127.1.237.195:46621/ using document root <none> and password file <none>
I20260927 13:43:54.488888 10558 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:54.489010 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:54.489302 10558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:43:54.491921 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "9afbb5e9862d44179217f1ac1d61a66c"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "296c5074a9a45185044885a945e20164"
server_key_iv: "8fe054278b2bac962b3e5d33a651ad53"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.492666 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "9afbb5e9862d44179217f1ac1d61a66c"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "296c5074a9a45185044885a945e20164"
server_key_iv: "8fe054278b2bac962b3e5d33a651ad53"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.497197 10558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:54.500231 10573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.501377 10558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:43:54.501545 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "9afbb5e9862d44179217f1ac1d61a66c"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "296c5074a9a45185044885a945e20164"
server_key_iv: "8fe054278b2bac962b3e5d33a651ad53"
server_key_version: "encryptionkey@0"
I20260927 13:43:54.501724 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:54.512432 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:54.515724 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:54.515985 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:54.516763 10558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:54.517869 10558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:54.517968 10558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.518108 10558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:54.518175 10558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:54.549355 10558 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:46827
I20260927 13:43:54.549432 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:46827 every 8 connection(s)
I20260927 13:43:54.550530 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:54.553129 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10558
I20260927 13:43:54.553292 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:43:54.553709 1975 external_mini_cluster.cc:1428] Setting key 03467a5e838e7baf2e62af836fc82b4e
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516634, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:43:54.568720 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.552836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:33505 (local address 127.1.237.254:42687)
0927 13:43:54.553034 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:54.553042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:54.554305 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 13:43:54.556056 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:54.556063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:54.556139 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:54.556253 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:54.557590 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.558306 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.559778 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.560169 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.560266 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.563090 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:54.563095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:54.563098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:54.563126 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:54.565744 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.566326 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.566329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.566331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.566434 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.566889 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.566893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.566896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.567235 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:54.567404 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:54.568199 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 13:43:54.568424 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:43:54.569543 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.553386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:33505)
0927 13:43:54.554139 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:54.554192 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:54.555822 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:54.556504 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:54.556520 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:54.556957 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:54.557436 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:54.557451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.558431 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.558434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:54.559626 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:54.559640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.560334 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.560338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:54.560513 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.561035 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:54.561053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:54.562935 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:43:54.565854 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:54.565860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:54.565869 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:54.566235 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:54.566520 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:43:54.566523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:43:54.566525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:43:54.566756 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:43:54.567368 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:43:54.567377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:43:54.567779 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 13:43:54.568086 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 13:43:54.568528 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":327,"threads_started":1}
I20260927 13:43:54.570600 10686 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42687
I20260927 13:43:54.570863 10686 heartbeater.cc:464] Registering TS with master...
I20260927 13:43:54.571623 10686 heartbeater.cc:511] Master 127.1.237.254:42687 requested a full tablet report, sending...
I20260927 13:43:54.572858 10230 ts_manager.cc:194] Registered new tserver with Master: 9afbb5e9862d44179217f1ac1d61a66c (127.1.237.195:46827)
I20260927 13:43:54.573616 10230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:33505
I20260927 13:43:54.580338 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
Entry for principal oryx/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab.
WARNING: no policy specified for oryx/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
Entry for principal oryx/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab.
I20260927 13:43:54.659081 10419 heartbeater.cc:503] Master 127.1.237.254:42687 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
Entry for principal oryx/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab.
I20260927 13:43:54.693518 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10291
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:43:54.708590 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.694526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.1.237.193:35299)
0927 13:43:54.695112 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:54.695119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:54.695146 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:43:54.695324 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:54.695331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:54.695556 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:54.695682 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:54.696893 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.697823 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.698953 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.699533 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.699623 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.703088 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:54.703097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:54.703111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:54.703143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:54.705881 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.706549 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.706559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.706563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.706688 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.707071 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.707076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.707080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.707467 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:54.707762 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:54.708022 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 13:43:54.708267 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":295,"threads_started":1}
I20260927 13:43:54.709707 10414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:54.786676 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10291
I20260927 13:43:54.803604 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10425
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:43:54.819031 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.804546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.1.237.194:44659)
0927 13:43:54.805204 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:54.805219 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:54.805244 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:43:54.805301 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:54.805307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:54.805515 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:54.805647 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:54.806920 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.807968 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.809237 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.809778 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.809879 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.814140 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:54.814150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:54.814163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:54.814193 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:54.816769 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.817312 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.817318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.817320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.817426 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.817732 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.817735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.817737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.818047 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:54.818259 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:54.818477 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 13:43:54.818644 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":312,"threads_started":1}
I20260927 13:43:54.820222 10548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:54.899734 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10425
I20260927 13:43:54.915136 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10558
Sep 27 13:43:54 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:43:54.930240 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:54.916062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.1.237.195:46827)
0927 13:43:54.916651 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:54.916658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:54.916684 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:43:54.916738 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:54.916743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:54.916940 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:54.917065 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:54.918165 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.919083 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.920346 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:54.920881 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:54.920981 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:54.925230 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:54.925240 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:54.925254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:54.925281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:54.927739 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.928318 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.928324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.928326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.928450 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:54.928787 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:54.928789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:54.928791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:54.929111 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:54.929397 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:54.929684 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 13:43:54.929866 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":297,"threads_started":1}
I20260927 13:43:54.931599 10681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:55.012580 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10558
I20260927 13:43:55.027788 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10198
I20260927 13:43:55.028980 10260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:43:55.123796 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10198
I20260927 13:43:55.143051 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42687
--webserver_interface=127.1.237.254
--webserver_port=39121
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:43:55.446913 10703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:55.447144 10703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:55.447330 10703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:55.454125 10703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:43:55.454237 10703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:55.454316 10703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:55.454397 10703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:43:55.454475 10703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:43:55.463624 10703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=39121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:55.464749 10703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:55.465942 10703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:55.476929 10709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:55.476850 10708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:55.476903 10711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:55.477742 10703 server_base.cc:1034] running on GCE node
I20260927 13:43:55.478420 10703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:55.479911 10703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:55.481109 10703 hybrid_clock.cc:648] HybridClock initialized: now 1790516635481064 us; error 38 us; skew 500 ppm
I20260927 13:43:55.481436 10703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:55 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516635, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:55.484403 10703 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:43:55.485646 10703 webserver.cc:460] Webserver started at http://127.1.237.254:39121/ using document root <none> and password file <none>
I20260927 13:43:55.486281 10703 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:55.486433 10703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:55.491333 10703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:43:55.493860 10718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:55.495040 10703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:43:55.495244 10703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "9c234e9340c24948833f5febb7613f95"
format_stamp: "Formatted at 2026-09-27 13:43:52 on dist-test-slave-7g3l"
server_key: "a9d551012e1935f73a4d0bee0c76b783"
server_key_iv: "8089f086319be2f84a4facc5e4399135"
server_key_version: "encryptionkey@0"
I20260927 13:43:55.496145 10703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:55.519647 10703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:55.523602 10703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:55.523856 10703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:55.540572 10703 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:42687
I20260927 13:43:55.540587 10769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:42687 every 8 connection(s)
I20260927 13:43:55.541882 10703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:43:55.548004 10770 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:43:55.550034 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10703
I20260927 13:43:55.550757 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:35299
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:42687
--webserver_port=35977
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:43:55.550910 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Bootstrap starting.
I20260927 13:43:55.573422 10770 log.cc:824] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Log is configured to *not* fsync() on all Append() calls
I20260927 13:43:55.583091 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:43:55.583647 10770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Bootstrap complete.
I20260927 13:43:55.595932 10770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:55.596928 10770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c234e9340c24948833f5febb7613f95, State: Initialized, Role: FOLLOWER
I20260927 13:43:55.597529 10770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:55.597759 10770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:43:55.597877 10770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:43:55.598032 10770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:43:55.603233 10770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:55.603772 10770 leader_election.cc:302] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c234e9340c24948833f5febb7613f95; no voters:
I20260927 13:43:55.604682 10770 leader_election.cc:288] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:43:55.604854 10774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:43:55.609793 10770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:43:55.612473 10774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [term 2 LEADER]: Becoming Leader. State: Replica: 9c234e9340c24948833f5febb7613f95, State: Running, Role: LEADER
I20260927 13:43:55.612913 10774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } }
I20260927 13:43:55.615249 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c234e9340c24948833f5febb7613f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } } }
I20260927 13:43:55.615581 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:55.617759 10776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c234e9340c24948833f5febb7613f95. Latest consensus state: current_term: 2 leader_uuid: "9c234e9340c24948833f5febb7613f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c234e9340c24948833f5febb7613f95" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42687 } } }
I20260927 13:43:55.618117 10782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:43:55.618254 10776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95 [sys.catalog]: This master's current role is: LEADER
I20260927 13:43:55.626916 10782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:43:55.628248 10782 catalog_manager.cc:1259] Loaded cluster ID: fc089d9499654160ba2802b1a870bb24
I20260927 13:43:55.628397 10782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:43:55.632213 10782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:43:55.633678 10782 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c234e9340c24948833f5febb7613f95: Loaded TSK: 0
I20260927 13:43:55.635200 10782 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:43:55.885689 10772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:55.885921 10772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:55.886065 10772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:55.893018 10772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:55.893105 10772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:55.893256 10772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:43:55.902274 10772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.193.keytab
--principal=oryx/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:35299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=35977
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:55.903488 10772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:55.904712 10772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:55.918962 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:55.919435 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:55.920467 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:55.920555 10772 server_base.cc:1034] running on GCE node
I20260927 13:43:55.921164 10772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:55.922055 10772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:55.923264 10772 hybrid_clock.cc:648] HybridClock initialized: now 1790516635923207 us; error 51 us; skew 500 ppm
I20260927 13:43:55.923552 10772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:55 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516635, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:55.926429 10772 init.cc:377] Logged in from keytab as oryx/127.1.237.193@KRBTEST.COM (short username oryx)
I20260927 13:43:55.927946 10772 webserver.cc:460] Webserver started at http://127.1.237.193:35977/ using document root <none> and password file <none>
I20260927 13:43:55.928663 10772 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:55.928795 10772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:55.933840 10772 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:43:55.936375 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:55.937675 10772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:43:55.937834 10772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "6a4f2616f90a4dc98e64e97427d7a84e"
format_stamp: "Formatted at 2026-09-27 13:43:53 on dist-test-slave-7g3l"
server_key: "171f2f780984ef0200cf282322b8f40e"
server_key_iv: "935e352bbf3709202363c1f78f48185e"
server_key_version: "encryptionkey@0"
I20260927 13:43:55.938655 10772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:55.957541 10772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:55.961269 10772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:55.961532 10772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:55.962431 10772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:55.963658 10772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:55.963748 10772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:55.963920 10772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:55.964160 10772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:55.995464 10772 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:35299
I20260927 13:43:55.995570 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:35299 every 8 connection(s)
I20260927 13:43:55.996796 10772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:43:56.002940 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10772
I20260927 13:43:56.003700 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:44659
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:42687
--webserver_port=41343
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:43:56 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:56 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:56.313584 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:56.313879 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:56.314060 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:56.320819 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:56.320942 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:56.321079 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:43:56.329596 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.194.keytab
--principal=oryx/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:44659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=41343
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:56.330830 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:56.332137 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:56.350349 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:56.350382 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:56.350593 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:56.352582 10923 server_base.cc:1034] running on GCE node
I20260927 13:43:56.353189 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:56.354105 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:56.355316 10923 hybrid_clock.cc:648] HybridClock initialized: now 1790516636355270 us; error 35 us; skew 500 ppm
I20260927 13:43:56.355628 10923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:56 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516636, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:56.358922 10923 init.cc:377] Logged in from keytab as oryx/127.1.237.194@KRBTEST.COM (short username oryx)
I20260927 13:43:56.360339 10923 webserver.cc:460] Webserver started at http://127.1.237.194:41343/ using document root <none> and password file <none>
I20260927 13:43:56.360984 10923 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:56.361138 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:56.366463 10923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:43:56.369352 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:56.370827 10923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:43:56.370981 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "87e13b3e1e794a32b16f6c8904e1cddd"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "12a5e9518c60822afbe0dbc21e7633f5"
server_key_iv: "4fc96a6877786e7299846f00cb7a9f08"
server_key_version: "encryptionkey@0"
I20260927 13:43:56.371933 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:56.388104 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:56.391993 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:56.392244 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:56.393167 10923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:56.394357 10923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:56.394477 10923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:56.394606 10923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:56.394690 10923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:56.434082 10923 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:44659
I20260927 13:43:56.434127 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:44659 every 8 connection(s)
I20260927 13:43:56.435729 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:43:56.437100 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 10923
I20260927 13:43:56.438128 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:46827
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:42687
--webserver_port=46621
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:43:56 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:56 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:57.145743 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:43:57.146158 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:43:57.146366 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:43:57.157757 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:43:57.157877 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:43:57.158053 11056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:43:57.173216 11056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/krb5kdc/oryx_127.1.237.195.keytab
--principal=oryx/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:46827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=46621
--tserver_master_addrs=127.1.237.254:42687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:43:57.174810 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:43:57.176488 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:43:57.199723 11062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:57.207680 11063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:43:57.208863 11065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:43:57.209201 11056 server_base.cc:1034] running on GCE node
I20260927 13:43:57.209896 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:43:57.211055 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:43:57.212266 11056 hybrid_clock.cc:648] HybridClock initialized: now 1790516637212212 us; error 54 us; skew 500 ppm
I20260927 13:43:57.212584 11056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516637, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:43:57.215700 11056 init.cc:377] Logged in from keytab as oryx/127.1.237.195@KRBTEST.COM (short username oryx)
I20260927 13:43:57.220327 11056 webserver.cc:460] Webserver started at http://127.1.237.195:46621/ using document root <none> and password file <none>
I20260927 13:43:57.221024 11056 fs_manager.cc:359] Metadata directory not provided
I20260927 13:43:57.221174 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:43:57.226295 11056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:43:57.228960 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:57.230391 11056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:43:57.230585 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "9afbb5e9862d44179217f1ac1d61a66c"
format_stamp: "Formatted at 2026-09-27 13:43:54 on dist-test-slave-7g3l"
server_key: "296c5074a9a45185044885a945e20164"
server_key_iv: "8fe054278b2bac962b3e5d33a651ad53"
server_key_version: "encryptionkey@0"
I20260927 13:43:57.231539 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:43:57.246572 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:43:57.250635 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:43:57.250887 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:43:57.251895 11056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:43:57.253400 11056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:43:57.253511 11056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:43:57.253630 11056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:43:57.253728 11056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260927 13:43:57.313118 10922 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.302s user 0.009s sys 0.008s
I20260927 13:43:57.315246 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:55.999662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:38417)
0927 13:43:56.000491 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:56.000533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:56.002151 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:56.003859 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:56.003872 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:56.004414 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:56.004913 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:56.004929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.007209 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.007219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:56.008331 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:56.008342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.009704 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.009708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:56.009908 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:56.010649 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:56.010670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.314105 (+1303435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":353,"threads_started":1}
W20260927 13:43:57.315285 10921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:55.999538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:38417 (local address 127.1.237.254:42687)
0927 13:43:56.000177 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:56.000188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:56.000645 (+ 457us) server_negotiation.cc:407] Connection header received
0927 13:43:56.002744 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:56.002781 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:56.003307 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:56.003733 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:56.005107 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.007026 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.008477 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.009513 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.009687 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.314491 (+1304804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:38417: BlockingRecv error: recv got EOF from 127.1.237.193:38417 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":247,"threads_started":1}
W20260927 13:43:57.316249 10919 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:42687 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.237.254:42687: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
I20260927 13:43:57.316522 11056 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:46827
I20260927 13:43:57.316563 11184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:46827 every 8 connection(s)
I20260927 13:43:57.318192 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:43:57.321085 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11056
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:57.334733 10921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:57.317193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:33315 (local address 127.1.237.254:42687)
0927 13:43:57.317340 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.317346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.317366 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:43:57.317558 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.317562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.317643 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.317765 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.318960 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.320571 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.322583 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.323529 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.323647 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.334376 (+ 10729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:33315: BlockingRecv error: recv got EOF from 127.1.237.193:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:43:57.335513 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.316882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:33315)
0927 13:43:57.317155 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:57.317183 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:57.317390 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:57.317822 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:57.317829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:57.318441 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:57.318728 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.318743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.320817 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.320822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.322384 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.322403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.323978 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.323987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.324240 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.324988 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:57.325019 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.335107 (+ 10088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:57.354784 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.336349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:58693)
0927 13:43:57.337084 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:57.337105 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:57.337218 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:57.338233 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:57.338240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:57.338739 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:57.338985 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.339000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.345122 (+ 6122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.345127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.346579 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.346591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.347683 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.347689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.347860 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.348607 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:57.348628 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.354485 (+ 5857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":597}
W20260927 13:43:57.355613 11190 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:57.336782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:58693 (local address 127.1.237.254:42687)
0927 13:43:57.337778 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.337786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.337806 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:43:57.337861 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.337868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.337962 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.339402 (+ 1440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.339740 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.341389 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.346745 (+ 5356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.347525 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.349276 (+ 1751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.355269 (+ 5993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:58693: BlockingRecv error: recv got EOF from 127.1.237.193:58693 (error 108)
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":841}
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:57.355705 10919 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:57.361771 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.322742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.1.237.254:42687)
0927 13:43:57.335005 (+ 12263us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.335013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.335910 (+ 897us) server_negotiation.cc:407] Connection header received
0927 13:43:57.336220 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.336229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.336356 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.336448 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.339407 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.341117 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.346489 (+ 5372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.347583 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.347740 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.352089 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:43:57.352104 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:43:57.352143 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:43:57.352207 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:43:57.357124 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:57.358027 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:57.358039 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:57.358052 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:57.358311 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:43:57.358764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:43:57.358771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:43:57.358777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:43:57.359645 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:43:57.360030 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 13:43:57.360573 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 13:43:57.361077 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":12134,"thread_start_us":15341,"threads_started":1}
W20260927 13:43:57.701661 11054 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.237s user 0.000s sys 0.015s
W20260927 13:43:57.702960 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:56.440805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:37325 (local address 127.1.237.254:42687)
0927 13:43:56.447530 (+ 6725us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:56.447539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:56.451216 (+ 3677us) server_negotiation.cc:407] Connection header received
0927 13:43:56.453836 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:56.453845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:56.453939 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:56.454062 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:56.457912 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.459450 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.462622 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.463493 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.463595 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.702619 (+1239024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:37325: BlockingRecv error: recv got EOF from 127.1.237.194:37325 (error 108)
Metrics: {"server-negotiator.queue_time_us":6577,"thread_start_us":6449,"threads_started":1}
I20260927 13:43:57.703533 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:56.440135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:37325)
0927 13:43:56.449799 (+ 9664us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:56.449858 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:56.453371 (+ 3513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:56.455618 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:56.455635 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:56.456647 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:56.457656 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:56.457684 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.459709 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.459717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:56.462414 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:56.462432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:56.463958 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:56.463963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:56.464190 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:56.464872 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:56.464907 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.702427 (+1237520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9238,"thread_start_us":13628,"threads_started":1}
W20260927 13:43:57.704378 11052 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:42687 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.237.254:42687: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:57.715389 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:57.705245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52311 (local address 127.1.237.254:42687)
0927 13:43:57.705411 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.705417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.705564 (+ 147us) server_negotiation.cc:407] Connection header received
0927 13:43:57.705786 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.705790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.705864 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.705965 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.706682 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.707488 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.708968 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.709405 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.709555 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.715214 (+ 5659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:52311: BlockingRecv error: recv got EOF from 127.1.237.194:52311 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:43:57.715698 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.705224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:52311)
0927 13:43:57.705481 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:57.705496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:57.705631 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:57.706001 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:57.706007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:57.706373 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:57.706539 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.706548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.707624 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.707630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.708796 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.708819 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.709764 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.709768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.709949 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.710732 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:57.710777 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.714990 (+ 4213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:57 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 13:43:57.728788 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:57.716876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:57477 (local address 127.1.237.254:42687)
0927 13:43:57.717049 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.717054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.717240 (+ 186us) server_negotiation.cc:407] Connection header received
0927 13:43:57.717433 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.717436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.717495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.717572 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.718762 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.719569 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.722282 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.722698 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.722790 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.728472 (+ 5682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:57477: BlockingRecv error: recv got EOF from 127.1.237.194:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:43:57.729075 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.716539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:57477)
0927 13:43:57.717129 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:57.717160 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:57.717314 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:57.717661 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:57.717669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:57.718248 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:57.718601 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.718623 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.719767 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.719775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.722109 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.722130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.722878 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.722887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.723205 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.723926 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:57.723959 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:57.728852 (+ 4893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433}
W20260927 13:43:57.729444 11052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:58.370828 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:58.356456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:33957)
0927 13:43:58.357248 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:58.357268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:58.357397 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:58.358038 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:58.358051 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:58.358572 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:58.358711 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.358719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.359788 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.359793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.360751 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.360762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.361482 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.361488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.361656 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.367001 (+ 5345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:58.367018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:58.370508 (+ 3490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":546,"threads_started":1}
W20260927 13:43:58.370843 11192 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:58.356478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:33957 (local address 127.1.237.254:42687)
0927 13:43:58.357178 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:58.357187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:58.357349 (+ 162us) server_negotiation.cc:407] Connection header received
0927 13:43:58.357530 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:58.357536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:58.357639 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:58.357879 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:58.358847 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.359649 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.360892 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.361310 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.361441 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.370678 (+ 9237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:33957: BlockingRecv error: recv got EOF from 127.1.237.193:33957 (error 108)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":358,"threads_started":1}
W20260927 13:43:58.446183 11188 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.092s user 0.006s sys 0.008s
W20260927 13:43:58.447583 11189 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:57.322494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:48619 (local address 127.1.237.254:42687)
0927 13:43:57.326842 (+ 4348us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:57.326849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:57.332157 (+ 5308us) server_negotiation.cc:407] Connection header received
0927 13:43:57.341646 (+ 9489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:57.341653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:57.341746 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:57.341833 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:57.345417 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.346560 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.348909 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.350728 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.350872 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.447326 (+1096454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:48619: BlockingRecv error: recv got EOF from 127.1.237.195:48619 (error 108)
Metrics: {"server-negotiator.queue_time_us":2341,"thread_start_us":2207,"threads_started":1}
I20260927 13:43:58.448293 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:57.321815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:48619)
0927 13:43:57.327474 (+ 5659us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:57.327539 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:57.331038 (+ 3499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:57.342575 (+ 11537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:57.342590 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:57.343609 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:57.344614 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.344645 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.346748 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.346752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.348751 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:57.348765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:57.351303 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:57.351314 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:57.351754 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:57.353833 (+ 2079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:57.353868 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:58.447049 (+1093181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5286,"thread_start_us":5053,"threads_started":1}
W20260927 13:43:58.449133 11185 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:42687 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.237.254:42687: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:58.457736 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:58.449738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:57013)
0927 13:43:58.449928 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:58.449949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:58.450059 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:58.450372 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:58.450377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:58.450738 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:58.450890 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.450899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.451949 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.451957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.452926 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.452939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.453619 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.453625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.453795 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.454343 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:58.454359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:58.457560 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:43:58.457893 11189 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:58.449799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:57013 (local address 127.1.237.254:42687)
0927 13:43:58.450046 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:58.450051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:58.450068 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:43:58.450163 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:58.450168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:58.450236 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:58.450345 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:58.451023 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.451815 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.453061 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.453456 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.453585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.457717 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:57013: BlockingRecv error: recv got EOF from 127.1.237.195:57013 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:58.466243 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:58.458371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:48659)
0927 13:43:58.458561 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:58.458580 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:58.458707 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:58.459113 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:58.459119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:58.459484 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:58.459669 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.459685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.460760 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.460763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.461680 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.461690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.462318 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.462321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.462443 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.462942 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:58.462958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:58.465984 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:43:58.466403 11189 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:58.458426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:48659 (local address 127.1.237.254:42687)
0927 13:43:58.458583 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:58.458589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:58.458628 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:43:58.458846 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:58.458851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:58.458947 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:58.459052 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:58.459806 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.460645 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.461794 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.462191 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.462290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.466152 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:48659: BlockingRecv error: recv got EOF from 127.1.237.195:48659 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 13:43:58.466526 11185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:58 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:58.738869 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:58.730003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:57051)
0927 13:43:58.730568 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:58.730583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:58.730694 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:58.730993 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:58.730996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:58.731482 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:58.731622 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.731630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.732625 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.732629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.733595 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:58.733606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.734294 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.734298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:58.734418 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.735090 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:58.735107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:58.738534 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":352,"threads_started":1}
W20260927 13:43:58.738978 11189 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:58.730031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:57051 (local address 127.1.237.254:42687)
0927 13:43:58.730201 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:58.730206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:58.730657 (+ 451us) server_negotiation.cc:407] Connection header received
0927 13:43:58.730784 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:58.730788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:58.730867 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:58.730946 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:58.731743 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.732481 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.733731 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:58.734143 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:58.734233 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:58.738732 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:57051: BlockingRecv error: recv got EOF from 127.1.237.194:57051 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:59.381160 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:59.371744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:43835)
0927 13:43:59.372262 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:59.372277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:59.372393 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:59.372688 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:59.372691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:59.373085 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:59.373253 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.373261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.374821 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.374826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.375908 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.375921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.376657 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.376662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.376777 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.377320 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:59.377338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:59.380779 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":303,"threads_started":1}
W20260927 13:43:59.381291 11196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:59.371773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:43835 (local address 127.1.237.254:42687)
0927 13:43:59.372237 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:59.372241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:59.372417 (+ 176us) server_negotiation.cc:407] Connection header received
0927 13:43:59.372480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:59.372483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:59.372558 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:59.372657 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:59.373383 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.374678 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.376080 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.376520 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.376617 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.381022 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:43835: BlockingRecv error: recv got EOF from 127.1.237.193:43835 (error 108)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":275,"threads_started":1}
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:59.476260 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:59.467061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:38445)
0927 13:43:59.467549 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:59.467564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:59.467672 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:59.467977 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:59.467981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:59.468392 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:59.468533 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.468541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.469553 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.469557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.470504 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.470515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.471267 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.471274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.471395 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.472022 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:59.472038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:59.476027 (+ 3989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":308,"threads_started":1}
W20260927 13:43:59.476539 11196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:59.467091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:38445 (local address 127.1.237.254:42687)
0927 13:43:59.467316 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:59.467326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:59.467644 (+ 318us) server_negotiation.cc:407] Connection header received
0927 13:43:59.467763 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:59.467767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:59.467824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:59.467914 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:59.468657 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.469408 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.470624 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.471032 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.471118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.476192 (+ 5074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:38445: BlockingRecv error: recv got EOF from 127.1.237.195:38445 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:43:59 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:43:59.748926 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:43:59.739984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:43435)
0927 13:43:59.740547 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 13:43:59.740563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:43:59.740711 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:43:59.741033 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:43:59.741038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:43:59.741512 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:43:59.741699 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.741707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.742707 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.742714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.743726 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:43:59.743741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.744499 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.744505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:43:59.744674 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.745265 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:43:59.745294 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:43:59.748599 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":365,"threads_started":1}
W20260927 13:43:59.749018 11196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:43:59.740043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43435 (local address 127.1.237.254:42687)
0927 13:43:59.740332 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 13:43:59.740341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:43:59.740643 (+ 302us) server_negotiation.cc:407] Connection header received
0927 13:43:59.740825 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:43:59.740831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:43:59.740901 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:43:59.741000 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:43:59.741828 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.742564 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.743869 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:43:59.744352 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:43:59.744485 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:43:59.748763 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:43435: BlockingRecv error: recv got EOF from 127.1.237.194:43435 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:00.390211 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:00.382349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:57839)
0927 13:44:00.382885 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:00.382901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:00.382995 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:00.383293 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:00.383296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:00.383604 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:00.383757 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.383766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.384702 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.384706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.385572 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.385582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.386162 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.386166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.386274 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.386820 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:00.386836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:00.389990 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":244,"threads_started":1}
W20260927 13:44:00.390385 11200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:00.382471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:57839 (local address 127.1.237.254:42687)
0927 13:44:00.382919 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:00.382924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:00.382940 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:44:00.383088 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:00.383092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:00.383171 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:00.383250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:00.383874 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.384588 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.385679 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.386056 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.386136 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.390158 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:57839: BlockingRecv error: recv got EOF from 127.1.237.193:57839 (error 108)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:00.485519 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:00.477090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:45619)
0927 13:44:00.477578 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:00.477593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:00.477702 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:00.477976 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:00.477979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:00.478329 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:00.478501 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.478509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.479494 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.479498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.480400 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.480412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.481028 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.481032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.481161 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.481696 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:00.481712 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:00.485189 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":286,"threads_started":1}
W20260927 13:44:00.485574 11200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:00.477160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45619 (local address 127.1.237.254:42687)
0927 13:44:00.477311 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:00.477315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:00.477663 (+ 348us) server_negotiation.cc:407] Connection header received
0927 13:44:00.477789 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:00.477793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:00.477845 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:00.477921 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:00.478614 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.479365 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.480511 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.480910 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.480993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.485347 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:45619: BlockingRecv error: recv got EOF from 127.1.237.195:45619 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:00 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:00.758183 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:00.749810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:52177)
0927 13:44:00.750354 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:00.750370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:00.750518 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:00.750834 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:00.750837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:00.751323 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:00.751508 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.751517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.752604 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.752609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.753508 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:00.753518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.754230 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.754235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:00.754409 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.754932 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:00.754948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:00.757883 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":339,"threads_started":1}
W20260927 13:44:00.758272 11200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:00.749944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52177 (local address 127.1.237.254:42687)
0927 13:44:00.750151 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:00.750157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:00.750449 (+ 292us) server_negotiation.cc:407] Connection header received
0927 13:44:00.750628 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:00.750634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:00.750691 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:00.750779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:00.751638 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.752462 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.753664 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:00.754075 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:00.754207 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:00.758028 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:52177: BlockingRecv error: recv got EOF from 127.1.237.194:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:01.400125 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:01.391150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:50391)
0927 13:44:01.391794 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:01.391813 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:01.391951 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:01.392291 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:01.392296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:01.392673 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:01.392826 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.392834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.393786 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.393793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.394784 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.394795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.395764 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.395770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.395970 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.396667 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:01.396684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:01.399771 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":346,"threads_started":1}
W20260927 13:44:01.400180 11204 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:01.391217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:50391 (local address 127.1.237.254:42687)
0927 13:44:01.391796 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:01.391803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:01.391883 (+ 80us) server_negotiation.cc:407] Connection header received
0927 13:44:01.392071 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:01.392079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:01.392145 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:01.392256 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:01.392976 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.393662 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.394929 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.395601 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.395725 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.399958 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:50391: BlockingRecv error: recv got EOF from 127.1.237.193:50391 (error 108)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":324,"threads_started":1}
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:01.495366 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:01.486409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:35743)
0927 13:44:01.486913 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:01.486928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:01.487032 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:01.487381 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:01.487384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:01.487722 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:01.487890 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.487898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.489077 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.489085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.490320 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.490334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.490942 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.490946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.491092 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.491833 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:01.491848 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:01.495011 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
W20260927 13:44:01.495451 11204 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:01.486433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:35743 (local address 127.1.237.254:42687)
0927 13:44:01.486622 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:01.486627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:01.486993 (+ 366us) server_negotiation.cc:407] Connection header received
0927 13:44:01.487130 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:01.487134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:01.487261 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:01.487341 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:01.488035 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.488919 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.490444 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.490832 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.490920 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.495229 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:35743: BlockingRecv error: recv got EOF from 127.1.237.195:35743 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:01 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:01.767318 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:01.759032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:55289)
0927 13:44:01.759523 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:01.759538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:01.759678 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:01.760024 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:01.760028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:01.760378 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:01.760525 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.760532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.761476 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.761482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.762384 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:01.762394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.763056 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.763061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:01.763246 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.763828 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:01.763843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:01.767016 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
W20260927 13:44:01.767473 11204 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:01.759094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:55289 (local address 127.1.237.254:42687)
0927 13:44:01.759323 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:01.759330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:01.759615 (+ 285us) server_negotiation.cc:407] Connection header received
0927 13:44:01.759796 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:01.759801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:01.759885 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:01.759982 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:01.760654 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.761354 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.762508 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:01.762925 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:01.763047 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:01.767262 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:55289: BlockingRecv error: recv got EOF from 127.1.237.194:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:02.409415 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:02.400922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:47147)
0927 13:44:02.401387 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:02.401403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:02.401500 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:02.401944 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:02.401946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:02.402309 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:02.402459 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.402467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.403907 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.403912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.404815 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.404825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.405414 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.405418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.405528 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.406036 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:02.406051 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:02.409109 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":225,"threads_started":1}
W20260927 13:44:02.409562 11208 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:02.400988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:47147 (local address 127.1.237.254:42687)
0927 13:44:02.401692 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:02.401696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:02.401711 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:44:02.401749 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:02.401752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:02.401814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:02.401900 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:02.402603 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.403765 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.404926 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.405297 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.405385 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.409286 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:47147: BlockingRecv error: recv got EOF from 127.1.237.193:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":493,"threads_started":1}
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:02.504494 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:02.496260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:45025)
0927 13:44:02.496711 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:02.496726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:02.496881 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:02.497227 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:02.497231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:02.497626 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:02.497780 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.497788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.498730 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.498737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.499800 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.499811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.500468 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.500472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.500654 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.501175 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:02.501191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:02.504253 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
W20260927 13:44:02.504617 11208 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:02.496316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45025 (local address 127.1.237.254:42687)
0927 13:44:02.496532 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:02.496540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:02.496808 (+ 268us) server_negotiation.cc:407] Connection header received
0927 13:44:02.497024 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:02.497029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:02.497094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:02.497201 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:02.497917 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.498605 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.499942 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.500338 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.500443 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.504428 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:45025: BlockingRecv error: recv got EOF from 127.1.237.195:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:02 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:02.776679 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:02.768331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:33243)
0927 13:44:02.768834 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:02.768848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:02.769005 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:02.769331 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:02.769335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:02.769748 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:02.769883 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.769891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.770780 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.770786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.771611 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:02.771621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.772297 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.772302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:02.772463 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.773084 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:02.773100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:02.776474 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
W20260927 13:44:02.776870 11208 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:02.768430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:33243 (local address 127.1.237.254:42687)
0927 13:44:02.768656 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:02.768665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:02.768943 (+ 278us) server_negotiation.cc:407] Connection header received
0927 13:44:02.769120 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:02.769125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:02.769204 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:02.769302 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:02.770010 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.770667 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.771753 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:02.772172 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:02.772289 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:02.776632 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:33243: BlockingRecv error: recv got EOF from 127.1.237.194:33243 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:03.427707 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:03.410288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:40125)
0927 13:44:03.410848 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:03.410883 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:03.410982 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:03.411511 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:03.411516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:03.411931 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:03.412091 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.412099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.413030 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.413034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.413970 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.413981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.414607 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.414611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.414725 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.424331 (+ 9606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:03.424349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:03.427480 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":297,"threads_started":1}
W20260927 13:44:03.427870 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:03.410561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:40125 (local address 127.1.237.254:42687)
0927 13:44:03.411181 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:03.411188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:03.411203 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:44:03.411253 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:03.411257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:03.411359 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:03.411460 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:03.412209 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.412921 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.414084 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.414488 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.414573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.427651 (+ 13078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:40125: BlockingRecv error: recv got EOF from 127.1.237.193:40125 (error 108)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":343,"threads_started":1}
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:03.513916 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:03.505359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:35667)
0927 13:44:03.505878 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:03.505893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:03.505999 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:03.506255 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:03.506258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:03.506666 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:03.506807 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.506816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.507910 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.507914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.508803 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.508814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.509443 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.509446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.509564 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.510144 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:03.510161 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:03.513646 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":284,"threads_started":1}
W20260927 13:44:03.514012 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:03.505444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:35667 (local address 127.1.237.254:42687)
0927 13:44:03.505597 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:03.505602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:03.505964 (+ 362us) server_negotiation.cc:407] Connection header received
0927 13:44:03.506076 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:03.506079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:03.506136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:03.506213 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:03.506961 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.507788 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.508928 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.509322 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.509418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.513817 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:35667: BlockingRecv error: recv got EOF from 127.1.237.195:35667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:03 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:03.785848 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:03.777484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:40043)
0927 13:44:03.777943 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:03.777957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:03.778116 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:03.778439 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:03.778443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:03.778832 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:03.778995 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.779003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.780022 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.780029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.780920 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:03.780930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.781580 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.781585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:03.781754 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.782298 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:03.782319 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:03.785622 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":258,"threads_started":1}
W20260927 13:44:03.786047 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:03.777544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:40043 (local address 127.1.237.254:42687)
0927 13:44:03.777764 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:03.777771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:03.778040 (+ 269us) server_negotiation.cc:407] Connection header received
0927 13:44:03.778230 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:03.778235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:03.778311 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:03.778410 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:03.779155 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.779891 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.781049 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:03.781437 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:03.781564 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:03.785782 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:40043: BlockingRecv error: recv got EOF from 127.1.237.194:40043 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:04.438151 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:04.428576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:56571)
0927 13:44:04.429280 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:04.429306 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:04.429440 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:04.429740 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:04.429743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:04.430165 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:04.430402 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.430414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.431489 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.431493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.432474 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.432485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.433104 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.433108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.433211 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.433812 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:04.433828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:04.437815 (+ 3987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":285,"threads_started":1}
W20260927 13:44:04.438283 11216 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:04.428663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:56571 (local address 127.1.237.254:42687)
0927 13:44:04.429483 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:04.429488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:04.429502 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:44:04.429559 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:04.429562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:04.429620 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:04.429703 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:04.430546 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.431368 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.432585 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.432985 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.433072 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.438032 (+ 4960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:56571: BlockingRecv error: recv got EOF from 127.1.237.193:56571 (error 108)
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":644,"threads_started":1}
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:04.523458 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:04.514908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:35261)
0927 13:44:04.515384 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:04.515401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:04.515517 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:04.515787 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:04.515789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:04.516186 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:04.516345 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.516353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.517193 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.517196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.518216 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.518229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.519217 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.519224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.519394 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.519949 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:04.519965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:04.523121 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":281,"threads_started":1}
W20260927 13:44:04.523535 11216 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:04.515002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:35261 (local address 127.1.237.254:42687)
0927 13:44:04.515194 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:04.515201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:04.515482 (+ 281us) server_negotiation.cc:407] Connection header received
0927 13:44:04.515605 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:04.515609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:04.515664 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:04.515752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:04.516450 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.517082 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.518365 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.519031 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.519176 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.523323 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:35261: BlockingRecv error: recv got EOF from 127.1.237.195:35261 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:04 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:04.795229 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:04.786804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:48729)
0927 13:44:04.787551 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:04.787570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:04.787722 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:04.788074 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:04.788078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:04.788464 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:04.788615 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.788622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.789608 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.789614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.790548 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:04.790558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.791217 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.791236 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:04.791392 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.791925 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:04.791941 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:04.794872 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":436,"threads_started":1}
W20260927 13:44:04.795305 11216 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:04.786885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:48729 (local address 127.1.237.254:42687)
0927 13:44:04.787121 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:04.787131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:04.787640 (+ 509us) server_negotiation.cc:407] Connection header received
0927 13:44:04.787856 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:04.787861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:04.787931 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:04.788027 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:04.788758 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.789472 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.790680 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:04.791050 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:04.791203 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:04.795046 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:48729: BlockingRecv error: recv got EOF from 127.1.237.194:48729 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:05.449527 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:05.439097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:35773)
0927 13:44:05.439668 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:05.439688 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:05.439808 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:05.440304 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:05.440309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:05.440820 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:05.441045 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.441060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.442656 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.442663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.444266 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.444282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.445126 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.445131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.445241 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.445885 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:05.445910 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:05.449306 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
W20260927 13:44:05.449733 11220 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:05.439213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:35773 (local address 127.1.237.254:42687)
0927 13:44:05.439959 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:05.439966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:05.439988 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:44:05.440047 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:05.440053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:05.440135 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:05.440256 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:05.441205 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.442416 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.444408 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.444983 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.445075 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.449485 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:35773: BlockingRecv error: recv got EOF from 127.1.237.193:35773 (error 108)
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":511,"threads_started":1}
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:05.533015 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:05.524449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:47601)
0927 13:44:05.524967 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:05.524983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:05.525088 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:05.525371 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:05.525374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:05.525726 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:05.525887 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.525896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.527006 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.527010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.528283 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.528295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.528918 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.528922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.529026 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.529550 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:05.529565 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:05.532749 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
W20260927 13:44:05.533138 11220 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:05.524611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:47601 (local address 127.1.237.254:42687)
0927 13:44:05.524874 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:05.524880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:05.525059 (+ 179us) server_negotiation.cc:407] Connection header received
0927 13:44:05.525177 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:05.525180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:05.525236 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:05.525304 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:05.526000 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.526860 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.528408 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.528785 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.528882 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.532922 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:47601: BlockingRecv error: recv got EOF from 127.1.237.195:47601 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:05 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:05.804585 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:05.796107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:56517)
0927 13:44:05.796634 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:05.796649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:05.796750 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:05.797034 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:05.797037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:05.797397 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:05.797547 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.797554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.798648 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.798651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.799685 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:05.799697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.800539 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.800546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:05.800708 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.801245 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:05.801261 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:05.804248 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":291,"threads_started":1}
W20260927 13:44:05.804698 11220 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:05.796238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:56517 (local address 127.1.237.254:42687)
0927 13:44:05.796436 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:05.796440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:05.796720 (+ 280us) server_negotiation.cc:407] Connection header received
0927 13:44:05.796847 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:05.796850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:05.796919 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:05.796998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:05.797659 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.798502 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.799836 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:05.800358 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:05.800505 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:05.804451 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:56517: BlockingRecv error: recv got EOF from 127.1.237.194:56517 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:06.459029 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:06.450417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:52287)
0927 13:44:06.450902 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:06.450917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:06.451026 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:06.451422 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:06.451425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:06.451820 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:06.451969 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.451978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.452897 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.452901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.453805 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.453815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.454455 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.454459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.454587 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.455217 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:06.455246 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:06.458674 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":260,"threads_started":1}
W20260927 13:44:06.459124 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:06.450470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52287 (local address 127.1.237.254:42687)
0927 13:44:06.450905 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:06.450909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:06.450993 (+ 84us) server_negotiation.cc:407] Connection header received
0927 13:44:06.451120 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:06.451123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:06.451247 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:06.451349 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:06.452090 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.452796 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.453924 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.454334 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.454422 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.458878 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:52287: BlockingRecv error: recv got EOF from 127.1.237.193:52287 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":247,"threads_started":1}
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:06.543303 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:06.533973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:49841)
0927 13:44:06.534481 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:06.534495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:06.534600 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:06.534923 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:06.534928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:06.535373 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:06.535614 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.535623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.536694 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.536699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.537942 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.537953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.538615 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.538619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.538723 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.539297 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:06.539313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:06.542941 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":265,"threads_started":1}
W20260927 13:44:06.543391 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:06.534023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:49841 (local address 127.1.237.254:42687)
0927 13:44:06.534164 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:06.534168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:06.534561 (+ 393us) server_negotiation.cc:407] Connection header received
0927 13:44:06.534688 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:06.534709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:06.534764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:06.534850 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:06.535725 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.536544 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.538076 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.538471 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.538564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.543116 (+ 4552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:49841: BlockingRecv error: recv got EOF from 127.1.237.195:49841 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:06 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:06.813787 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:06.805432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:41337)
0927 13:44:06.805913 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:06.805947 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:06.806069 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:06.806348 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:06.806351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:06.806710 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:06.806861 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.806869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.807893 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.807897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.808802 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:06.808812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.809417 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.809421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:06.809539 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.810088 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:06.810103 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:06.813454 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
W20260927 13:44:06.813944 11224 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:06.805506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:41337 (local address 127.1.237.254:42687)
0927 13:44:06.805735 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:06.805741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:06.806031 (+ 290us) server_negotiation.cc:407] Connection header received
0927 13:44:06.806159 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:06.806162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:06.806230 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:06.806305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:06.806992 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.807757 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.808912 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:06.809301 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:06.809386 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:06.813640 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:41337: BlockingRecv error: recv got EOF from 127.1.237.194:41337 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:07.468710 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:07.460012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:51313)
0927 13:44:07.460463 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:07.460479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:07.460574 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:07.461021 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:07.461024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:07.461390 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:07.461534 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.461542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.462483 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.462487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.463502 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.463512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.464154 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.464158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.464262 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.464846 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:07.464861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:07.468479 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
W20260927 13:44:07.468894 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:07.460051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:51313 (local address 127.1.237.254:42687)
0927 13:44:07.460758 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:07.460762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:07.460777 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:44:07.460822 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:07.460825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:07.460891 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:07.460981 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:07.461662 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.462352 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.463614 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.464016 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.464105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.468644 (+ 4539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:51313: BlockingRecv error: recv got EOF from 127.1.237.193:51313 (error 108)
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":516,"threads_started":1}
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:07.553861 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:07.544312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:43473)
0927 13:44:07.544822 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:07.544841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:07.544967 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:07.545284 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:07.545290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:07.545770 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:07.545955 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.545965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.547511 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.547517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.549004 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.549023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.549715 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.549721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.549894 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.550511 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:07.550535 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:07.553644 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
W20260927 13:44:07.554015 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:07.544356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:43473 (local address 127.1.237.254:42687)
0927 13:44:07.544578 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:07.544588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:07.544928 (+ 340us) server_negotiation.cc:407] Connection header received
0927 13:44:07.545068 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:07.545072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:07.545131 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:07.545207 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:07.546134 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.547356 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.549153 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.549572 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.549665 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.553813 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:43473: BlockingRecv error: recv got EOF from 127.1.237.195:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:07 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:07.823469 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:07.814822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:45225)
0927 13:44:07.815569 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:07.815584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:07.815730 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:07.816070 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:07.816074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:07.816443 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:07.816595 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.816603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.817582 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.817591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.818561 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:07.818572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.819421 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.819427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:07.819602 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.820226 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:07.820244 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:07.823261 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":514,"threads_started":1}
W20260927 13:44:07.823632 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:07.814904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:45225 (local address 127.1.237.254:42687)
0927 13:44:07.815140 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:07.815173 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:07.815670 (+ 497us) server_negotiation.cc:407] Connection header received
0927 13:44:07.815852 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:07.815857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:07.815942 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:07.816038 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:07.816739 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.817453 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.818700 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:07.819262 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:07.819410 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:07.823394 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:45225: BlockingRecv error: recv got EOF from 127.1.237.194:45225 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:08.484640 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:08.469488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:60455)
0927 13:44:08.469901 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:08.469916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:08.470005 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:08.470413 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:08.470416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:08.470769 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:08.471025 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.471038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.472069 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.472076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.473110 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.473120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.473806 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.473811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.473989 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.480916 (+ 6927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:08.480933 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:08.484307 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":228,"threads_started":1}
W20260927 13:44:08.484833 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:08.469720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:60455 (local address 127.1.237.254:42687)
0927 13:44:08.470158 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:08.470162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:08.470175 (+ 13us) server_negotiation.cc:407] Connection header received
0927 13:44:08.470212 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:08.470215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:08.470272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:08.470380 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:08.471178 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.471938 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.473241 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.473667 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.473796 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.484524 (+ 10728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:60455: BlockingRecv error: recv got EOF from 127.1.237.193:60455 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:08.562420 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:08.554655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:45357)
0927 13:44:08.555207 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:08.555228 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:08.555345 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:08.555619 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:08.555622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:08.555964 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:08.556127 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.556136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.557018 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.557022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.557879 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.557889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.558490 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.558494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.558595 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.559256 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:08.559274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:08.562207 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
W20260927 13:44:08.562542 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:08.554712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45357 (local address 127.1.237.254:42687)
0927 13:44:08.554865 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:08.554869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:08.555308 (+ 439us) server_negotiation.cc:407] Connection header received
0927 13:44:08.555437 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:08.555440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:08.555495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:08.555574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:08.556243 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.556910 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.558012 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.558377 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.558459 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.562365 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:45357: BlockingRecv error: recv got EOF from 127.1.237.195:45357 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:08 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:08.832772 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:08.824278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:46521)
0927 13:44:08.824757 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:08.824772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:08.824877 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:08.825174 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:08.825177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:08.825566 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:08.825740 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.825747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.826643 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.826646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.827558 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:08.827568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.828453 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.828459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:08.828644 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.829254 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:08.829281 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:08.832573 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
W20260927 13:44:08.832983 11232 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:08.824368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:46521 (local address 127.1.237.254:42687)
0927 13:44:08.824537 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:08.824541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:08.824839 (+ 298us) server_negotiation.cc:407] Connection header received
0927 13:44:08.824968 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:08.824972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:08.825039 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:08.825112 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:08.825851 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.826544 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.827677 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:08.828277 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:08.828439 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:08.832728 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:46521: BlockingRecv error: recv got EOF from 127.1.237.194:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:09.494225 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:09.485554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:42173)
0927 13:44:09.486036 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:09.486052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:09.486161 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:09.486644 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:09.486647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:09.486971 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:09.487130 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.487138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.488133 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.488137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.489150 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.489162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.489890 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.489895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.490092 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.490640 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:09.490658 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:09.493998 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
W20260927 13:44:09.494383 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:09.485615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:42173 (local address 127.1.237.254:42687)
0927 13:44:09.486392 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:09.486397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:09.486410 (+ 13us) server_negotiation.cc:407] Connection header received
0927 13:44:09.486450 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:09.486453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:09.486508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:09.486587 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:09.487290 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.488013 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.489305 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.489741 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.489880 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.494181 (+ 4301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:42173: BlockingRecv error: recv got EOF from 127.1.237.193:42173 (error 108)
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":577,"threads_started":1}
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:09.571236 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:09.563251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:58077)
0927 13:44:09.563661 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:09.563677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:09.563787 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:09.564052 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:09.564055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:09.564415 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:09.564589 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.564597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.565516 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.565520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.566430 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.566440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.567053 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.567057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.567227 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.567737 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:09.567753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:09.571030 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":235,"threads_started":1}
W20260927 13:44:09.571455 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:09.563325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:58077 (local address 127.1.237.254:42687)
0927 13:44:09.563535 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:09.563542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:09.563754 (+ 212us) server_negotiation.cc:407] Connection header received
0927 13:44:09.563879 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:09.563882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:09.563938 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:09.564011 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:09.564698 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.565411 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.566550 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.566933 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.567022 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.571216 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:58077: BlockingRecv error: recv got EOF from 127.1.237.195:58077 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:09 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:09.842449 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:09.833590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:43917)
0927 13:44:09.834130 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:09.834146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:09.834283 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:09.834629 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:09.834632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:09.835073 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:09.835359 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.835376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.836751 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.836757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.837654 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:09.837665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.838344 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.838348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:09.838464 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.838983 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:09.838998 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:09.842229 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":290,"threads_started":1}
W20260927 13:44:09.842654 11236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:09.833613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43917 (local address 127.1.237.254:42687)
0927 13:44:09.833757 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:09.833761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:09.834210 (+ 449us) server_negotiation.cc:407] Connection header received
0927 13:44:09.834430 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:09.834434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:09.834501 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:09.834575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:09.835589 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.836541 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.837780 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:09.838195 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:09.838286 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:09.842436 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:43917: BlockingRecv error: recv got EOF from 127.1.237.194:43917 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:10.503930 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:10.495068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:52947)
0927 13:44:10.495579 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:10.495595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:10.495688 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:10.495992 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:10.495995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:10.496371 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:10.496512 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.496521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.497425 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.497428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.498247 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.498256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.498959 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.498965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.499148 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.499910 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:10.499935 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:10.503606 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
W20260927 13:44:10.504026 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:10.495142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52947 (local address 127.1.237.254:42687)
0927 13:44:10.495633 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:10.495637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:10.495652 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:44:10.495809 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:10.495813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:10.495872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:10.495955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:10.496626 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.497301 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.498398 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.498811 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.498943 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.503780 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:52947: BlockingRecv error: recv got EOF from 127.1.237.193:52947 (error 108)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":255,"threads_started":1}
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:10.580046 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:10.572021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:45757)
0927 13:44:10.572602 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:10.572617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:10.572718 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:10.572990 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:10.572993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:10.573369 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:10.573530 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.573538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.574451 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.574454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.575452 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.575464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.576119 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.576124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.576305 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.576827 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:10.576842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:10.579805 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":326,"threads_started":1}
W20260927 13:44:10.580205 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:10.572082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45757 (local address 127.1.237.254:42687)
0927 13:44:10.572234 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:10.572238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:10.572683 (+ 445us) server_negotiation.cc:407] Connection header received
0927 13:44:10.572802 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:10.572805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:10.572862 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:10.572937 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:10.573637 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.574337 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.575564 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.575959 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.576046 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.579986 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:45757: BlockingRecv error: recv got EOF from 127.1.237.195:45757 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:10 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:10.852119 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:10.843428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:42913)
0927 13:44:10.844001 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:10.844015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:10.844159 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:10.844481 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:10.844486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:10.844948 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:10.845087 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.845095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.846068 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.846074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.846970 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:10.846980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.847928 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.847934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:10.848094 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.848666 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:10.848690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:10.851763 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
W20260927 13:44:10.852191 11240 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:10.843614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:42913 (local address 127.1.237.254:42687)
0927 13:44:10.843910 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:10.843916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:10.844098 (+ 182us) server_negotiation.cc:407] Connection header received
0927 13:44:10.844273 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:10.844284 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:10.844358 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:10.844454 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:10.845225 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.845939 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.847108 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:10.847792 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:10.847915 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:10.851970 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:42913: BlockingRecv error: recv got EOF from 127.1.237.194:42913 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.193@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:11.512997 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:11.504946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.193:39367)
0927 13:44:11.505411 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:11.505426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:11.505521 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:11.505985 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:11.505988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:11.506320 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:11.506478 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.506486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.507452 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.507455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.508340 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.508350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.508958 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.508962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.509076 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.509585 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:11.509600 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:11.512778 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":239,"threads_started":1}
W20260927 13:44:11.513299 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:11.504995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:39367 (local address 127.1.237.254:42687)
0927 13:44:11.505720 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:11.505724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:11.505738 (+ 14us) server_negotiation.cc:407] Connection header received
0927 13:44:11.505796 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:11.505799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:11.505863 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:11.505943 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:11.506591 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.507347 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.508444 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.508842 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.508925 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.512949 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.193:39367: BlockingRecv error: recv got EOF from 127.1.237.193:39367 (error 108)
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":533,"threads_started":1}
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.195@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:11.589332 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:11.580919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.195:39411)
0927 13:44:11.581369 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:11.581384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:11.581529 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:11.581843 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:11.581847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:11.582261 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:11.582387 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.582394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.583491 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.583497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.584359 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.584369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.585026 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.585032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.585199 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.585722 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:11.585737 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:11.589112 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
W20260927 13:44:11.589509 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:11.580978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:39411 (local address 127.1.237.254:42687)
0927 13:44:11.581237 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:11.581243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:11.581467 (+ 224us) server_negotiation.cc:407] Connection header received
0927 13:44:11.581641 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:11.581648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:11.581706 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:11.581813 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:11.582534 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.583353 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.584511 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.584893 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.585004 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.589286 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.195:39411: BlockingRecv error: recv got EOF from 127.1.237.195:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 13:44:11 dist-test-slave-7g3l krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.237.194@KRBTEST.COM for oryx/127.1.237.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 13:44:11.861871 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:11.853291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42687 (local address 127.1.237.194:43893)
0927 13:44:11.853767 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:11.853782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:11.853931 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:11.854285 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:11.854289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:11.854684 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:11.854878 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.854887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.855941 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.855948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.856853 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:11.856863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.857619 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.857625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:11.857801 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.858299 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:11.858315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:11.861645 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.237.254:42687: Server oryx/127.1.237.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
W20260927 13:44:11.862018 11244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:11.853373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43893 (local address 127.1.237.254:42687)
0927 13:44:11.853571 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:11.853579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:11.853867 (+ 288us) server_negotiation.cc:407] Connection header received
0927 13:44:11.854043 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:11.854047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:11.854122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:11.854230 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:11.855013 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.855804 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.857000 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:11.857468 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:11.857599 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:11.861819 (+ 4220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.237.194:43893: BlockingRecv error: recv got EOF from 127.1.237.194:43893 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:44:12.321575 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10772
I20260927 13:44:12.338228 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:12.322593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.1.237.193:35299)
0927 13:44:12.323114 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:12.323120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:12.323143 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:44:12.323222 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:12.323228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:12.323474 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:12.323627 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:12.324808 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.325805 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.326875 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.327400 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.327492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:12.335110 (+ 7618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:12.335119 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:12.335132 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:12.335235 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:12.337690 (+ 2455us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:44:12.337984 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35522: Request ticket server kudu/127.1.237.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":266,"threads_started":1}
W20260927 13:44:12.338346 11248 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35522: Request ticket server kudu/127.1.237.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:44:12.338333 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10772: Not authorized: Client connection negotiation failed: client connection to 127.1.237.193:35299: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:44:12.338546 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10772
I20260927 13:44:12.355250 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10923
I20260927 13:44:12.368716 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:12.356197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.1.237.194:44659)
0927 13:44:12.356795 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:12.356802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:12.356829 (+ 27us) server_negotiation.cc:407] Connection header received
0927 13:44:12.356887 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:12.356893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:12.357128 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:12.357285 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:12.359017 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.360112 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.361231 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.361744 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.361866 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:12.364130 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:12.364137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:12.364150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:12.364180 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:12.367869 (+ 3689us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:44:12.368434 (+ 565us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36368: Request ticket server kudu/127.1.237.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
W20260927 13:44:12.368824 11249 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36368: Request ticket server kudu/127.1.237.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:44:12.369306 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10923: Not authorized: Client connection negotiation failed: client connection to 127.1.237.194:44659: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:44:12.369467 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10923
I20260927 13:44:12.391680 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11056
I20260927 13:44:12.411924 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:12.397359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.1.237.195:46827)
0927 13:44:12.397920 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:12.397927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:12.397953 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:44:12.398021 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:12.398027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:12.398267 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:12.398501 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:12.400151 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.401628 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.403655 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:12.404474 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:12.404600 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:12.408142 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:12.408153 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:12.408171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:12.408225 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:12.411355 (+ 3130us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 13:44:12.411605 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47828: Request ticket server kudu/127.1.237.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":263,"threads_started":1}
W20260927 13:44:12.412019 11250 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47828: Request ticket server kudu/127.1.237.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 13:44:12.411994 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11056: Not authorized: Client connection negotiation failed: client connection to 127.1.237.195:46827: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.237.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 13:44:12.412155 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11056
I20260927 13:44:12.428418 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 10703
I20260927 13:44:12.429592 10765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:12.539950 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 10703
2026-09-27T13:44:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21152 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T13:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:44:12Z Disabled control of system clock
I20260927 13:44:12.609023 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42945
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 13:44:12.910507 11255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:12.910746 11255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:12.910869 11255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:12.917481 11255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:44:12.917557 11255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:12.917637 11255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:12.917727 11255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:44:12.917784 11255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:44:12.926296 11255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:42945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:12.927434 11255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:12.928992 11255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:12.940003 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:12.940043 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:12.940194 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:12.940090 11255 server_base.cc:1034] running on GCE node
I20260927 13:44:12.940982 11255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:12.942145 11255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:12.943354 11255 hybrid_clock.cc:648] HybridClock initialized: now 1790516652943319 us; error 29 us; skew 500 ppm
I20260927 13:44:12.943653 11255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:12.945626 11255 webserver.cc:460] Webserver started at http://127.1.237.254:39041/ using document root <none> and password file <none>
I20260927 13:44:12.946259 11255 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:12.946393 11255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:12.946681 11255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:12.949277 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "3f564d0dcbc248a68984dc4f8601daae"
format_stamp: "Formatted at 2026-09-27 13:44:12 on dist-test-slave-7g3l"
server_key: "b055593ed56b7a47b6f386c918f6d763"
server_key_iv: "80a43556276a48d10b78f55d051c1f25"
server_key_version: "encryptionkey@0"
I20260927 13:44:12.949988 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "3f564d0dcbc248a68984dc4f8601daae"
format_stamp: "Formatted at 2026-09-27 13:44:12 on dist-test-slave-7g3l"
server_key: "b055593ed56b7a47b6f386c918f6d763"
server_key_iv: "80a43556276a48d10b78f55d051c1f25"
server_key_version: "encryptionkey@0"
I20260927 13:44:12.954417 11255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 13:44:12.957487 11269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:12.958593 11255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:44:12.958747 11255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "3f564d0dcbc248a68984dc4f8601daae"
format_stamp: "Formatted at 2026-09-27 13:44:12 on dist-test-slave-7g3l"
server_key: "b055593ed56b7a47b6f386c918f6d763"
server_key_iv: "80a43556276a48d10b78f55d051c1f25"
server_key_version: "encryptionkey@0"
I20260927 13:44:12.958949 11255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:12.984663 11255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:12.985463 11255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:12.985734 11255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:13.003219 11255 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:42945
I20260927 13:44:13.003309 11320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:42945 every 8 connection(s)
I20260927 13:44:13.004472 11255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:44:13.005002 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11255
I20260927 13:44:13.005191 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:44:13.005966 1975 external_mini_cluster.cc:1428] Setting key 9a7f7314ff41506d9cd9ace332dcfd49
I20260927 13:44:13.008569 11321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:13.018748 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:13.007957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.1.237.254:42945)
0927 13:44:13.008895 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:13.008905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:13.008944 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:44:13.010418 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:13.010435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:13.010854 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:13.011149 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:13.011785 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.013006 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.014752 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.015386 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.015505 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:13.016697 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:13.016706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:13.016718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:13.017217 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:13.017288 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:13.017317 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:13.017399 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:13.017616 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 13:44:13.017927 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":344,"threads_started":1}
I20260927 13:44:13.019804 11321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae: Bootstrap starting.
I20260927 13:44:13.022954 11321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:13.023952 11321 log.cc:824] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:13.026500 11321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae: No bootstrap required, opened a new log
I20260927 13:44:13.032910 11321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } }
I20260927 13:44:13.033188 11321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:13.033255 11321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f564d0dcbc248a68984dc4f8601daae, State: Initialized, Role: FOLLOWER
I20260927 13:44:13.033702 11321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } }
I20260927 13:44:13.033872 11321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:44:13.033991 11321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:44:13.034116 11321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:13.036361 11321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } }
I20260927 13:44:13.036761 11321 leader_election.cc:302] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f564d0dcbc248a68984dc4f8601daae; no voters:
I20260927 13:44:13.037251 11321 leader_election.cc:288] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:44:13.037395 11326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:44:13.038051 11326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [term 1 LEADER]: Becoming Leader. State: Replica: 3f564d0dcbc248a68984dc4f8601daae, State: Running, Role: LEADER
I20260927 13:44:13.038590 11326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } }
I20260927 13:44:13.039117 11321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:44:13.041254 11327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f564d0dcbc248a68984dc4f8601daae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } } }
I20260927 13:44:13.041702 11327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:13.041368 11328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f564d0dcbc248a68984dc4f8601daae. Latest consensus state: current_term: 1 leader_uuid: "3f564d0dcbc248a68984dc4f8601daae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f564d0dcbc248a68984dc4f8601daae" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 42945 } } }
I20260927 13:44:13.042184 11328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:13.043488 11335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:44:13.048066 11335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:44:13.055497 11335 catalog_manager.cc:1347] Generated new cluster ID: b5044b53a5974b04aafc4fa657e847fe
I20260927 13:44:13.055598 11335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:44:13.074656 11335 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:44:13.076196 11335 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:44:13.082049 11335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f564d0dcbc248a68984dc4f8601daae: Generated new TSK 0
I20260927 13:44:13.082808 11335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:44:13.090283 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--builtin_ntp_servers=127.1.237.212:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 13:44:13.410425 11345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:13.410770 11345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:13.410916 11345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:13.418700 11345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:13.418782 11345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:13.418924 11345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:44:13.427459 11345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:13.428637 11345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:13.430284 11345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:13.444525 11350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:13.444674 11351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:13.445060 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:13.446061 11345 server_base.cc:1034] running on GCE node
I20260927 13:44:13.446748 11345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:13.447669 11345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:13.448863 11345 hybrid_clock.cc:648] HybridClock initialized: now 1790516653448830 us; error 31 us; skew 500 ppm
I20260927 13:44:13.449170 11345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:13.451243 11345 webserver.cc:460] Webserver started at http://127.1.237.193:42965/ using document root <none> and password file <none>
I20260927 13:44:13.451908 11345 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:13.452014 11345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:13.452345 11345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:13.454914 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "20a6d0d51f444eb281ab35d14403afe7"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "e6a90e2ff4258fb19caaf946ae081413"
server_key_iv: "64d158b530b090fecfc480002cc27e34"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.455792 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "20a6d0d51f444eb281ab35d14403afe7"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "e6a90e2ff4258fb19caaf946ae081413"
server_key_iv: "64d158b530b090fecfc480002cc27e34"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.460453 11345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:44:13.463796 11359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.464965 11345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:44:13.465116 11345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "20a6d0d51f444eb281ab35d14403afe7"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "e6a90e2ff4258fb19caaf946ae081413"
server_key_iv: "64d158b530b090fecfc480002cc27e34"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.465356 11345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:13.496841 11345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:13.497603 11345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:13.497886 11345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:13.498795 11345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:13.500092 11345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:13.500198 11345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.500327 11345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:13.500411 11345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.532110 11345 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:44459
I20260927 13:44:13.532197 11471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:44459 every 8 connection(s)
I20260927 13:44:13.533409 11345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:44:13.535650 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11345
I20260927 13:44:13.535840 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:44:13.536651 1975 external_mini_cluster.cc:1428] Setting key cc832405de0fa59bb680d36c84223e39
I20260927 13:44:13.539144 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--builtin_ntp_servers=127.1.237.212:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:44:13.545845 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:13.535892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:50051 (local address 127.1.237.254:42945)
0927 13:44:13.536370 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:13.536377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:13.537641 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 13:44:13.539466 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:13.539472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:13.539546 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:13.539658 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:13.540804 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.541532 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.543209 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.543605 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.543738 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:13.545043 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:13.545051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:13.545061 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:13.545076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:13.545116 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:13.545128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:13.545217 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:13.545441 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 13:44:13.545527 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260927 13:44:13.546739 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:13.536509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42945 (local address 127.1.237.193:50051)
0927 13:44:13.537500 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:13.537550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:13.539119 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:13.539891 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:13.539899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:13.539979 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:13.540656 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:13.540674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.541670 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.541675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:13.543025 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:13.543036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.543824 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.543833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:13.544055 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:13.544730 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:13.544732 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:13.544767 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:13.544837 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:13.545216 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:13.545228 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:13.545238 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 13:44:13.545433 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 13:44:13.545777 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":316,"threads_started":1}
I20260927 13:44:13.548120 11472 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42945
I20260927 13:44:13.548409 11472 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:13.549224 11472 heartbeater.cc:511] Master 127.1.237.254:42945 requested a full tablet report, sending...
I20260927 13:44:13.551126 11286 ts_manager.cc:194] Registered new tserver with Master: 20a6d0d51f444eb281ab35d14403afe7 (127.1.237.193:44459)
I20260927 13:44:13.552558 11286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.193:50051
W20260927 13:44:13.845875 11476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:13.846120 11476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:13.846266 11476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:13.853026 11476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:13.853106 11476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:13.853278 11476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:44:13.861863 11476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:13.863071 11476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:13.864766 11476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:13.876955 11482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:13.876959 11484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:13.877111 11481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:13.877556 11476 server_base.cc:1034] running on GCE node
I20260927 13:44:13.878109 11476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:13.878877 11476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:13.880069 11476 hybrid_clock.cc:648] HybridClock initialized: now 1790516653880040 us; error 29 us; skew 500 ppm
I20260927 13:44:13.880347 11476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:13.882268 11476 webserver.cc:460] Webserver started at http://127.1.237.194:42425/ using document root <none> and password file <none>
I20260927 13:44:13.882894 11476 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:13.883006 11476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:13.883363 11476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:13.885890 11476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "f3b9da96eb2a48979ee98fe6fd3b7ce8"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "460bf11f8ae6a3179deede268798b8bb"
server_key_iv: "e367f35cfc5ae8f28291836632049072"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.886608 11476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "f3b9da96eb2a48979ee98fe6fd3b7ce8"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "460bf11f8ae6a3179deede268798b8bb"
server_key_iv: "e367f35cfc5ae8f28291836632049072"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.891032 11476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:44:13.894143 11490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.895398 11476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:44:13.895555 11476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "f3b9da96eb2a48979ee98fe6fd3b7ce8"
format_stamp: "Formatted at 2026-09-27 13:44:13 on dist-test-slave-7g3l"
server_key: "460bf11f8ae6a3179deede268798b8bb"
server_key_iv: "e367f35cfc5ae8f28291836632049072"
server_key_version: "encryptionkey@0"
I20260927 13:44:13.895754 11476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:13.909157 11476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:13.909862 11476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:13.910149 11476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:13.910984 11476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:13.912300 11476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:13.912408 11476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.912529 11476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:13.912617 11476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:13.943349 11476 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:46275
I20260927 13:44:13.943418 11602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:46275 every 8 connection(s)
I20260927 13:44:13.944442 11476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:44:13.949361 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11476
I20260927 13:44:13.949507 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:44:13.950027 1975 external_mini_cluster.cc:1428] Setting key 6c21db35a0cc893db7c4f40cadb29291
I20260927 13:44:13.952883 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--builtin_ntp_servers=127.1.237.212:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 13:44:13.957152 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:13.946604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:56255 (local address 127.1.237.254:42945)
0927 13:44:13.946757 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:13.946761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:13.948193 (+ 1432us) server_negotiation.cc:407] Connection header received
0927 13:44:13.950362 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:13.950366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:13.950428 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:13.950510 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:13.951768 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.952696 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.954196 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.954857 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.954999 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:13.956290 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:13.956298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:13.956306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:13.956328 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:13.956369 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:13.956385 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:13.956479 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:13.956680 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 13:44:13.956785 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:44:13.957813 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:13.947085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42945 (local address 127.1.237.194:56255)
0927 13:44:13.947969 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:13.948009 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:13.950072 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:13.950729 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:13.950738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:13.950841 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:13.951527 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:13.951547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.952853 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.952856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:13.954057 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:13.954068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:13.955076 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:13.955080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:13.955311 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:13.955984 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:13.955986 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:13.956020 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:13.956082 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:13.956480 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:13.956487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:13.956492 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:44:13.956609 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 13:44:13.956875 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":252,"threads_started":1}
I20260927 13:44:13.959043 11603 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42945
I20260927 13:44:13.959364 11603 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:13.960093 11603 heartbeater.cc:511] Master 127.1.237.254:42945 requested a full tablet report, sending...
I20260927 13:44:13.961449 11286 ts_manager.cc:194] Registered new tserver with Master: f3b9da96eb2a48979ee98fe6fd3b7ce8 (127.1.237.194:46275)
I20260927 13:44:13.962560 11286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.194:56255
W20260927 13:44:14.262516 11606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:14.262743 11606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:14.262817 11606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:14.269428 11606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:14.269507 11606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:14.269665 11606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:44:14.278347 11606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:14.279429 11606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:14.281003 11606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:14.293094 11612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:14.293133 11611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:14.293165 11614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:14.293890 11606 server_base.cc:1034] running on GCE node
I20260927 13:44:14.294356 11606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:14.295101 11606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:14.296332 11606 hybrid_clock.cc:648] HybridClock initialized: now 1790516654296282 us; error 47 us; skew 500 ppm
I20260927 13:44:14.296618 11606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:14.298527 11606 webserver.cc:460] Webserver started at http://127.1.237.195:35051/ using document root <none> and password file <none>
I20260927 13:44:14.299224 11606 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:14.299345 11606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:14.299631 11606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:14.302026 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "bad80cb2a45e4fadba3f147a0b8ee168"
format_stamp: "Formatted at 2026-09-27 13:44:14 on dist-test-slave-7g3l"
server_key: "9706a0d10f59d437b6556dad95536784"
server_key_iv: "6ae05ffe6cdd9b529f60dccd77ffced6"
server_key_version: "encryptionkey@0"
I20260927 13:44:14.302729 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "bad80cb2a45e4fadba3f147a0b8ee168"
format_stamp: "Formatted at 2026-09-27 13:44:14 on dist-test-slave-7g3l"
server_key: "9706a0d10f59d437b6556dad95536784"
server_key_iv: "6ae05ffe6cdd9b529f60dccd77ffced6"
server_key_version: "encryptionkey@0"
I20260927 13:44:14.307137 11606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:44:14.310053 11620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:14.311117 11606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 13:44:14.311286 11606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "bad80cb2a45e4fadba3f147a0b8ee168"
format_stamp: "Formatted at 2026-09-27 13:44:14 on dist-test-slave-7g3l"
server_key: "9706a0d10f59d437b6556dad95536784"
server_key_iv: "6ae05ffe6cdd9b529f60dccd77ffced6"
server_key_version: "encryptionkey@0"
I20260927 13:44:14.311487 11606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:14.336777 11606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:14.337498 11606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:14.337766 11606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:14.338593 11606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:14.340062 11606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:14.340178 11606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:14.340307 11606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:14.340394 11606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:14.371045 11606 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:33395
I20260927 13:44:14.371115 11732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:33395 every 8 connection(s)
I20260927 13:44:14.372217 11606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:44:14.373648 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11606
I20260927 13:44:14.373826 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:44:14.374341 1975 external_mini_cluster.cc:1428] Setting key bd2c8afb2573fe1d9c7f4787bf794dae
I20260927 13:44:14.382767 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:14.374403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:40741 (local address 127.1.237.254:42945)
0927 13:44:14.374547 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:14.374551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:14.376177 (+ 1626us) server_negotiation.cc:407] Connection header received
0927 13:44:14.377722 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:14.377726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:14.377788 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:14.377863 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:14.378710 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.379507 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.380778 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.381107 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.381191 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:14.382207 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:14.382211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:14.382214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:14.382228 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:14.382257 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:14.382267 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:14.382327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:14.382490 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 13:44:14.382549 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:44:14.383665 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:14.375038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:42945 (local address 127.1.237.195:40741)
0927 13:44:14.376028 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:14.376074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:14.377510 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:14.378002 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:14.378009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:14.378090 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:14.378580 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:14.378594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.379616 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.379619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:14.380675 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:14.380685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.381263 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.381267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:14.381437 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:14.381957 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:14.381959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:14.381989 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:14.382050 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:14.382339 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:14.382346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:14.382351 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:44:14.382472 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 13:44:14.382686 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":311,"threads_started":1}
I20260927 13:44:14.384898 11733 heartbeater.cc:347] Connected to a master server at 127.1.237.254:42945
I20260927 13:44:14.385154 11733 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:14.385792 11733 heartbeater.cc:511] Master 127.1.237.254:42945 requested a full tablet report, sending...
I20260927 13:44:14.387004 11286 ts_manager.cc:194] Registered new tserver with Master: bad80cb2a45e4fadba3f147a0b8ee168 (127.1.237.195:33395)
I20260927 13:44:14.388108 11286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.195:40741
I20260927 13:44:14.389065 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:44:14.394574 11474 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:14.393345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.1.237.254:42945)
0927 13:44:14.393504 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:14.393509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:14.393908 (+ 399us) server_negotiation.cc:407] Connection header received
0927 13:44:14.394013 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:14.394016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:14.394075 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:14.394152 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:14.394440 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36812: BlockingRecv error: recv got EOF from 127.0.0.1:36812 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:44:14.395470 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11345
I20260927 13:44:14.402127 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:14.396273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.1.237.193:44459)
0927 13:44:14.396780 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:14.396785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:14.396808 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:44:14.396857 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:14.396862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:14.397023 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:14.397134 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:14.397423 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.398434 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.399676 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.400158 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.400251 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:14.401078 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:14.401084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:14.401093 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:14.401557 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:14.401604 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:14.401620 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:14.401701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:14.401790 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 13:44:14.401876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":256,"threads_started":1}
I20260927 13:44:14.403105 11467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:14.484330 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11345
I20260927 13:44:14.500617 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11476
I20260927 13:44:14.508028 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:14.501422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.1.237.194:46275)
0927 13:44:14.501890 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:14.501897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:14.501922 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:44:14.501982 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:14.501988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:14.502185 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:14.502359 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:14.502737 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.503759 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.504928 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.505462 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.505595 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:14.506578 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:14.506598 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:14.506612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:14.507217 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:14.507283 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:14.507302 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:14.507418 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:14.507569 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 13:44:14.507707 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260927 13:44:14.509177 11598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:14.586918 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11476
I20260927 13:44:14.602396 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11606
I20260927 13:44:14.610132 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:14.603317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.1.237.195:33395)
0927 13:44:14.603873 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:14.603880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:14.603909 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:44:14.603981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:14.603987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:14.604184 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:14.604348 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:14.604766 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.605767 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.607021 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:14.607653 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:14.607784 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:14.608791 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:14.608801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:14.608814 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:14.609344 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:14.609385 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:14.609403 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:14.609533 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:14.609676 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 13:44:14.609789 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
I20260927 13:44:14.611310 11728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:14.688288 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11606
I20260927 13:44:14.703941 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11255
I20260927 13:44:14.705080 11316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:14.800743 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11255
2026-09-27T13:44:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2255 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T13:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:44:14Z Disabled control of system clock
I20260927 13:44:14.866644 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36625
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:44:15.167552 11752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:15.167795 11752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:15.167938 11752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:15.175266 11752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:44:15.175348 11752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:15.175429 11752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:15.175527 11752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:44:15.175590 11752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:44:15.184129 11752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:15.185222 11752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:15.186852 11752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:15.198125 11760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:15.198194 11757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:15.198215 11758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:15.199273 11752 server_base.cc:1034] running on GCE node
I20260927 13:44:15.200060 11752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:15.201452 11752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:15.202679 11752 hybrid_clock.cc:648] HybridClock initialized: now 1790516655202640 us; error 49 us; skew 500 ppm
I20260927 13:44:15.203110 11752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:15.205454 11752 webserver.cc:460] Webserver started at http://127.1.237.254:46321/ using document root <none> and password file <none>
I20260927 13:44:15.206157 11752 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:15.206297 11752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:15.206645 11752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:15.209491 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "cfc675da427b44389e49b3f4d31d113a"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "38dc39023dc40e293958092d6a7a5df9"
server_key_iv: "a1100482e348eb4c042d0eefa031ef9e"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.210297 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "cfc675da427b44389e49b3f4d31d113a"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "38dc39023dc40e293958092d6a7a5df9"
server_key_iv: "a1100482e348eb4c042d0eefa031ef9e"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.215626 11752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 13:44:15.218889 11766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:15.220240 11752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:44:15.220430 11752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "cfc675da427b44389e49b3f4d31d113a"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "38dc39023dc40e293958092d6a7a5df9"
server_key_iv: "a1100482e348eb4c042d0eefa031ef9e"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.220616 11752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:15.226039 11752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:15.226745 11752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:15.227020 11752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:15.243499 11752 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36625
I20260927 13:44:15.243520 11817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36625 every 8 connection(s)
I20260927 13:44:15.244788 11752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:44:15.248389 11818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:15.253201 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11752
I20260927 13:44:15.253397 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:44:15.253969 1975 external_mini_cluster.cc:1428] Setting key 12f6132817ee240313722307405077d3
I20260927 13:44:15.258644 11818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a: Bootstrap starting.
I20260927 13:44:15.262108 11818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:15.262233 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:15.255815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.1.237.254:36625)
0927 13:44:15.256796 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:15.256809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:15.256862 (+ 53us) server_negotiation.cc:407] Connection header received
0927 13:44:15.258409 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:15.258429 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:15.258817 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:15.259225 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:15.260238 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:15.260244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:15.260255 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:15.260268 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:15.260324 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:15.260343 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:15.260567 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:15.260696 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 13:44:15.260909 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":341,"threads_started":1}
I20260927 13:44:15.263394 11818 log.cc:824] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:15.266045 11818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a: No bootstrap required, opened a new log
I20260927 13:44:15.273223 11818 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } }
I20260927 13:44:15.273526 11818 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:15.273641 11818 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc675da427b44389e49b3f4d31d113a, State: Initialized, Role: FOLLOWER
I20260927 13:44:15.274209 11818 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } }
I20260927 13:44:15.274420 11818 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:44:15.274538 11818 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:44:15.274703 11818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:15.276710 11818 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } }
I20260927 13:44:15.277177 11818 leader_election.cc:302] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfc675da427b44389e49b3f4d31d113a; no voters:
I20260927 13:44:15.277710 11818 leader_election.cc:288] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:44:15.277841 11823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:44:15.278517 11823 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [term 1 LEADER]: Becoming Leader. State: Replica: cfc675da427b44389e49b3f4d31d113a, State: Running, Role: LEADER
I20260927 13:44:15.279033 11823 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } }
I20260927 13:44:15.279618 11818 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:44:15.281569 11825 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfc675da427b44389e49b3f4d31d113a. Latest consensus state: current_term: 1 leader_uuid: "cfc675da427b44389e49b3f4d31d113a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } } }
I20260927 13:44:15.281893 11825 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:15.282399 11824 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfc675da427b44389e49b3f4d31d113a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc675da427b44389e49b3f4d31d113a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36625 } } }
I20260927 13:44:15.282722 11824 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:15.282749 11828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:44:15.286597 11828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:44:15.306175 11828 catalog_manager.cc:1347] Generated new cluster ID: 8895a8d769bb42f399d83c6d50b4953a
I20260927 13:44:15.306288 11828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:44:15.315231 11828 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:44:15.315665 11828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:44:15.341473 11828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfc675da427b44389e49b3f4d31d113a: Generated new TSK 0
I20260927 13:44:15.342384 11828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:44:15.352425 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--builtin_ntp_servers=127.1.237.212:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 13:44:15.666652 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:15.666901 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:15.667042 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:15.673939 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:15.674016 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:15.674192 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:44:15.682595 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:15.683736 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:15.685338 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:15.697328 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:15.697237 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:15.697283 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:15.698055 11842 server_base.cc:1034] running on GCE node
I20260927 13:44:15.698621 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:15.699474 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:15.700676 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790516655700635 us; error 53 us; skew 500 ppm
I20260927 13:44:15.700958 11842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:15.702885 11842 webserver.cc:460] Webserver started at http://127.1.237.193:34281/ using document root <none> and password file <none>
I20260927 13:44:15.703560 11842 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:15.703680 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:15.703974 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:15.708069 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "7634f52da0e246199c2991e94be2cde0"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "866a5e1c3f12904cc8602435458e4e69"
server_key_iv: "18a14ed1c7a7f7ce20e56b75cc44aaa5"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.708772 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "7634f52da0e246199c2991e94be2cde0"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "866a5e1c3f12904cc8602435458e4e69"
server_key_iv: "18a14ed1c7a7f7ce20e56b75cc44aaa5"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.713326 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:44:15.716385 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:15.717551 11842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:44:15.717708 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "7634f52da0e246199c2991e94be2cde0"
format_stamp: "Formatted at 2026-09-27 13:44:15 on dist-test-slave-7g3l"
server_key: "866a5e1c3f12904cc8602435458e4e69"
server_key_iv: "18a14ed1c7a7f7ce20e56b75cc44aaa5"
server_key_version: "encryptionkey@0"
I20260927 13:44:15.717912 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:15.722831 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:15.723753 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:15.723968 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:15.724843 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:15.726003 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:15.726086 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:15.726157 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:15.726207 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:15.756080 11842 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37923
I20260927 13:44:15.756160 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37923 every 8 connection(s)
I20260927 13:44:15.757308 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:44:15.763870 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11842
I20260927 13:44:15.764089 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:44:15.764847 1975 external_mini_cluster.cc:1428] Setting key ac4074361538ba66e24a0e1f6fa46443
I20260927 13:44:15.765165 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:15.759624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52575 (local address 127.1.237.254:36625)
0927 13:44:15.759824 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:15.759848 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:15.761330 (+ 1482us) server_negotiation.cc:407] Connection header received
0927 13:44:15.762882 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:15.762886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:15.762936 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:15.763034 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:15.764346 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:15.764353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:15.764359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:15.764381 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:15.764420 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:15.764436 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:15.764565 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:15.764847 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 13:44:15.764907 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:44:15.765975 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:15.760185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36625 (local address 127.1.237.193:52575)
0927 13:44:15.761166 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:15.761221 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:15.762662 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:15.763272 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:15.763280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:15.763364 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:15.764022 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:15.764025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:15.764075 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:15.764153 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:15.764527 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:15.764534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:15.764539 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:44:15.764728 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 13:44:15.764936 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":260,"threads_started":1}
I20260927 13:44:15.767501 11969 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36625
I20260927 13:44:15.767763 11969 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:15.767747 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--builtin_ntp_servers=127.1.237.212:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:44:15.768261 11969 heartbeater.cc:511] Master 127.1.237.254:36625 requested a full tablet report, sending...
I20260927 13:44:15.770198 11783 ts_manager.cc:194] Registered new tserver with Master: 7634f52da0e246199c2991e94be2cde0 (127.1.237.193:37923)
W20260927 13:44:16.072013 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:16.072257 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:16.072448 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:16.079523 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:16.079602 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:16.079746 11972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:44:16.088044 11972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:16.089123 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:16.090796 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:16.103395 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:16.103420 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:16.103452 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:16.104084 11972 server_base.cc:1034] running on GCE node
I20260927 13:44:16.104638 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:16.105523 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:16.106737 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790516656106703 us; error 47 us; skew 500 ppm
I20260927 13:44:16.107028 11972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:16.109207 11972 webserver.cc:460] Webserver started at http://127.1.237.194:35623/ using document root <none> and password file <none>
I20260927 13:44:16.109856 11972 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:16.109978 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:16.110283 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:16.112922 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "b461e9159a0d4b379ef0d95ee10499a4"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "5f926028f09b1dd0f9b3aa79b2649425"
server_key_iv: "e226fd1202149afe9d8b11c17e5c2160"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.113649 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "b461e9159a0d4b379ef0d95ee10499a4"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "5f926028f09b1dd0f9b3aa79b2649425"
server_key_iv: "e226fd1202149afe9d8b11c17e5c2160"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.118249 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:44:16.121531 11986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.122694 11972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:44:16.122886 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "b461e9159a0d4b379ef0d95ee10499a4"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "5f926028f09b1dd0f9b3aa79b2649425"
server_key_iv: "e226fd1202149afe9d8b11c17e5c2160"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.123080 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:16.128314 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:16.129011 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:16.129256 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:16.130050 11972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:16.131304 11972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:16.131412 11972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.131531 11972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:16.131621 11972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.162060 11972 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:43091
I20260927 13:44:16.162137 12098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:43091 every 8 connection(s)
I20260927 13:44:16.163311 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:44:16.167176 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 11972
I20260927 13:44:16.167361 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:44:16.167761 1975 external_mini_cluster.cc:1428] Setting key 75b84a02dab137fad3998053984ebe0f
I20260927 13:44:16.170557 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--builtin_ntp_servers=127.1.237.212:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 13:44:16.171059 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.165635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:42669 (local address 127.1.237.254:36625)
0927 13:44:16.165811 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.165815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.167116 (+ 1301us) server_negotiation.cc:407] Connection header received
0927 13:44:16.168901 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.168905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.168953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.169033 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.170300 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:16.170308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:16.170314 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:16.170337 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:16.170379 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:16.170395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:16.170525 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:16.170671 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 13:44:16.170734 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:44:16.171868 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.166127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36625 (local address 127.1.237.194:42669)
0927 13:44:16.166974 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:16.167008 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:16.168628 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:16.169274 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:16.169287 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:16.169388 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:16.170018 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:16.170020 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:16.170054 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:16.170129 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:16.170490 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:16.170497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:16.170502 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:44:16.170739 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 13:44:16.170975 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":251,"threads_started":1}
I20260927 13:44:16.173094 12099 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36625
I20260927 13:44:16.173375 12099 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:16.174026 12099 heartbeater.cc:511] Master 127.1.237.254:36625 requested a full tablet report, sending...
I20260927 13:44:16.175413 11783 ts_manager.cc:194] Registered new tserver with Master: b461e9159a0d4b379ef0d95ee10499a4 (127.1.237.194:43091)
W20260927 13:44:16.477161 12102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:16.477413 12102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:16.477504 12102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:16.484447 12102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:16.484524 12102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:16.484613 12102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:44:16.492817 12102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:16.493918 12102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:16.495548 12102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:16.507426 12108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:16.507450 12107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:16.507555 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:16.508384 12102 server_base.cc:1034] running on GCE node
I20260927 13:44:16.508929 12102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:16.509814 12102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:16.510989 12102 hybrid_clock.cc:648] HybridClock initialized: now 1790516656510953 us; error 35 us; skew 500 ppm
I20260927 13:44:16.511320 12102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:44:16.513315 12102 webserver.cc:460] Webserver started at http://127.1.237.195:39945/ using document root <none> and password file <none>
I20260927 13:44:16.513938 12102 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:16.514045 12102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:16.514365 12102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:16.516920 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "213607043a784d1bbbb1f45b4b0dc16e"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "80a25648f819acea2a6f102fcfdd595e"
server_key_iv: "f8d1e895ae43681d1d4817aa2f2dc934"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.517624 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "213607043a784d1bbbb1f45b4b0dc16e"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "80a25648f819acea2a6f102fcfdd595e"
server_key_iv: "f8d1e895ae43681d1d4817aa2f2dc934"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.522313 12102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 13:44:16.525498 12116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.526729 12102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:44:16.526933 12102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "213607043a784d1bbbb1f45b4b0dc16e"
format_stamp: "Formatted at 2026-09-27 13:44:16 on dist-test-slave-7g3l"
server_key: "80a25648f819acea2a6f102fcfdd595e"
server_key_iv: "f8d1e895ae43681d1d4817aa2f2dc934"
server_key_version: "encryptionkey@0"
I20260927 13:44:16.527206 12102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:16.532248 12102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:16.532958 12102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:16.533219 12102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:16.533998 12102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:16.535291 12102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:16.535405 12102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.535534 12102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:16.535629 12102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:16.565564 12102 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:37621
I20260927 13:44:16.565637 12228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:37621 every 8 connection(s)
I20260927 13:44:16.566727 12102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:44:16.570937 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 12102
I20260927 13:44:16.571097 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:44:16.571485 1975 external_mini_cluster.cc:1428] Setting key aa887c62d23386c000453a05e5f77374
I20260927 13:44:16.574116 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.569001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:34757 (local address 127.1.237.254:36625)
0927 13:44:16.569210 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.569216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.570409 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 13:44:16.572030 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.572035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.572109 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.572214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.573454 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:16.573459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:16.573464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:16.573481 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:16.573517 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:16.573528 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:16.573634 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:16.573824 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 13:44:16.573885 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:44:16.574928 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.569499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36625 (local address 127.1.237.195:34757)
0927 13:44:16.570283 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:16.570316 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:16.571767 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:16.572420 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:16.572427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:16.572507 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:16.573167 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:44:16.573173 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:16.573221 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:44:16.573307 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:16.573623 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:16.573632 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:16.573640 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 13:44:16.573902 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 13:44:16.574119 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260927 13:44:16.576040 12229 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36625
I20260927 13:44:16.576313 12229 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:16.576817 12229 heartbeater.cc:511] Master 127.1.237.254:36625 requested a full tablet report, sending...
I20260927 13:44:16.577953 11783 ts_manager.cc:194] Registered new tserver with Master: 213607043a784d1bbbb1f45b4b0dc16e (127.1.237.195:37621)
I20260927 13:44:16.585841 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 13:44:16.591365 11821 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:44:16.589958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.1.237.254:36625)
0927 13:44:16.590124 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.590132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.590598 (+ 466us) server_negotiation.cc:407] Connection header received
0927 13:44:16.590702 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.590705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.590747 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.590827 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.591139 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52148: BlockingRecv error: recv got EOF from 127.0.0.1:52148 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:44:16.592145 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11842
I20260927 13:44:16.595319 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.593026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.1.237.193:37923)
0927 13:44:16.593521 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.593527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.593548 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:44:16.593608 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.593613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.593772 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.593878 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.594630 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:16.594635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:16.594643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:16.594657 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:16.594689 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:16.594701 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:16.594797 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:16.594912 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 13:44:16.594974 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":264,"threads_started":1}
I20260927 13:44:16.596326 11964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:16.671960 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11842
I20260927 13:44:16.687661 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11972
I20260927 13:44:16.691228 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.688416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.1.237.194:43091)
0927 13:44:16.688942 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.688949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.688975 (+ 26us) server_negotiation.cc:407] Connection header received
0927 13:44:16.689033 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.689039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.689244 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.689473 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.690463 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:16.690474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:16.690490 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:16.690506 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:16.690558 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:16.690573 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:16.690710 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:16.690823 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 13:44:16.690920 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":269,"threads_started":1}
I20260927 13:44:16.692368 12094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:16.769918 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11972
I20260927 13:44:16.784747 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 12102
I20260927 13:44:16.788519 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:16.785610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.1.237.195:37621)
0927 13:44:16.786185 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:16.786191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:16.786226 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:44:16.786302 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:16.786308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:16.786517 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:16.786677 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:16.787829 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:16.787835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:16.787847 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:44:16.787863 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:16.787899 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:44:16.787911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:16.788087 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:16.788156 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 13:44:16.788244 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":297,"threads_started":1}
I20260927 13:44:16.789706 12224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:16.868158 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 12102
I20260927 13:44:16.884006 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 11752
I20260927 13:44:16.885099 11813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:16.982587 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 11752
2026-09-27T13:44:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2182 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:44:17 dist-test-slave-7g3l krb5kdc[12247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:44:17 dist-test-slave-7g3l krb5kdc[12247](info): set up 2 sockets
Sep 27 13:44:17 dist-test-slave-7g3l krb5kdc[12247](info): commencing operation
krb5kdc: starting...
W20260927 13:44:18.036008 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:44:18 dist-test-slave-7g3l krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:44:18.227711 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:39295
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:44:18.526015 12260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:18.526266 12260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:18.526433 12260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:18.533267 12260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:44:18.533373 12260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:18.533478 12260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:18.533571 12260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:44:18.533635 12260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:44:18.542562 12260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:18.543745 12260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:18.544920 12260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:18.556249 12265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:18.556221 12268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:18.556159 12266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:18.556819 12260 server_base.cc:1034] running on GCE node
I20260927 13:44:18.557541 12260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:18.558713 12260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:18.559932 12260 hybrid_clock.cc:648] HybridClock initialized: now 1790516658559895 us; error 47 us; skew 500 ppm
I20260927 13:44:18.560233 12260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:44:18 dist-test-slave-7g3l krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:44:18.569253 12260 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:44:18.570603 12260 webserver.cc:460] Webserver started at http://127.1.237.254:34239/ using document root <none> and password file <none>
I20260927 13:44:18.571285 12260 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:18.571417 12260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:18.571696 12260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:18.574342 12260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "c832e3952eea4f54bf5f5165ebcbc0a2"
format_stamp: "Formatted at 2026-09-27 13:44:18 on dist-test-slave-7g3l"
server_key: "63def563bd9cef8560c21961dbd5441c"
server_key_iv: "814d54cc9b5bf4ba6a1fd7472138da09"
server_key_version: "encryptionkey@0"
I20260927 13:44:18.575021 12260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "c832e3952eea4f54bf5f5165ebcbc0a2"
format_stamp: "Formatted at 2026-09-27 13:44:18 on dist-test-slave-7g3l"
server_key: "63def563bd9cef8560c21961dbd5441c"
server_key_iv: "814d54cc9b5bf4ba6a1fd7472138da09"
server_key_version: "encryptionkey@0"
I20260927 13:44:18.579578 12260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:44:18.582587 12275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:18.583856 12260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:44:18.584028 12260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "c832e3952eea4f54bf5f5165ebcbc0a2"
format_stamp: "Formatted at 2026-09-27 13:44:18 on dist-test-slave-7g3l"
server_key: "63def563bd9cef8560c21961dbd5441c"
server_key_iv: "814d54cc9b5bf4ba6a1fd7472138da09"
server_key_version: "encryptionkey@0"
I20260927 13:44:18.584189 12260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:18.613790 12260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:18.617202 12260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:18.617460 12260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:18.634428 12260 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:39295
I20260927 13:44:18.634467 12326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:39295 every 8 connection(s)
I20260927 13:44:18.635686 12260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:44:18.638938 12327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:18.643946 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 12260
I20260927 13:44:18.644168 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:44:18.644922 1975 external_mini_cluster.cc:1428] Setting key 49f4df4997b6c5af4ae8334bf1ff6e36
I20260927 13:44:18.649442 12327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: Bootstrap starting.
I20260927 13:44:18.652825 12327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:18.653720 12327 log.cc:824] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:18.656368 12327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: No bootstrap required, opened a new log
Sep 27 13:44:18 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:44:18.666146 12327 raft_consensus.cc:348] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } }
I20260927 13:44:18.666543 12327 raft_consensus.cc:374] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:18.666648 12327 raft_consensus.cc:729] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c832e3952eea4f54bf5f5165ebcbc0a2, State: Initialized, Role: FOLLOWER
I20260927 13:44:18.666883 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:18.646847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.1.237.254:39295)
0927 13:44:18.647649 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:18.647659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:18.647698 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:44:18.649204 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:18.649224 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:18.649605 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:18.649904 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:18.651645 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:18.652905 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:18.654269 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:18.655072 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:18.655250 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:18.659020 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:18.659030 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:18.659046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:18.659111 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:18.662834 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:18.663438 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:18.663446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:18.663453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:18.663668 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:18.664025 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:18.664030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:18.664034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:18.664692 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:18.664977 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:18.665249 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 13:44:18.665634 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":424,"threads_started":1}
I20260927 13:44:18.667284 12327 consensus_queue.cc:260] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } }
I20260927 13:44:18.667515 12327 raft_consensus.cc:388] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:44:18.667640 12327 raft_consensus.cc:482] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:44:18.667850 12327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:18.669921 12327 raft_consensus.cc:504] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } }
I20260927 13:44:18.670364 12327 leader_election.cc:302] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c832e3952eea4f54bf5f5165ebcbc0a2; no voters:
I20260927 13:44:18.670898 12327 leader_election.cc:288] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:44:18.671023 12332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:44:18.671769 12332 raft_consensus.cc:686] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [term 1 LEADER]: Becoming Leader. State: Replica: c832e3952eea4f54bf5f5165ebcbc0a2, State: Running, Role: LEADER
I20260927 13:44:18.672230 12332 consensus_queue.cc:237] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } }
I20260927 13:44:18.672883 12327 sys_catalog.cc:564] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:44:18.674837 12333 sys_catalog.cc:455] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } } }
I20260927 13:44:18.676049 12333 sys_catalog.cc:458] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:18.675800 12334 sys_catalog.cc:455] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c832e3952eea4f54bf5f5165ebcbc0a2. Latest consensus state: current_term: 1 leader_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c832e3952eea4f54bf5f5165ebcbc0a2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 39295 } } }
I20260927 13:44:18.676159 12334 sys_catalog.cc:458] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:44:18.678140 12340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:44:18.682292 12340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:44:18.690303 12340 catalog_manager.cc:1347] Generated new cluster ID: b4d0ad03afe14af382ac25759d6d1e01
I20260927 13:44:18.690418 12340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:44:18.702195 12340 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:44:18.704011 12340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:44:18.716773 12340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: Generated new TSK 0
I20260927 13:44:18.717792 12340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:44:18.768249 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:44:19.078998 12353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:19.079309 12353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:19.079406 12353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:19.086436 12353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:19.086522 12353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:19.086661 12353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:44:19.095959 12353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:19.097074 12353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:19.098273 12353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:19.110666 12358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:19.110697 12359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:19.110756 12361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:19.111375 12353 server_base.cc:1034] running on GCE node
I20260927 13:44:19.111912 12353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:19.112677 12353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:19.113893 12353 hybrid_clock.cc:648] HybridClock initialized: now 1790516659113847 us; error 67 us; skew 500 ppm
I20260927 13:44:19.114184 12353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:44:19 dist-test-slave-7g3l krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:44:19.117105 12353 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:44:19.118399 12353 webserver.cc:460] Webserver started at http://127.1.237.193:39217/ using document root <none> and password file <none>
I20260927 13:44:19.118961 12353 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:19.119052 12353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:19.119347 12353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:19.122119 12353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "bead919308d14384af5b62af8cd67cc6"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "e5906136e9d3242679ca20470b15b3aa"
server_key_iv: "d5e7f24392f43c84549e55fa962922ab"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.122769 12353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "bead919308d14384af5b62af8cd67cc6"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "e5906136e9d3242679ca20470b15b3aa"
server_key_iv: "d5e7f24392f43c84549e55fa962922ab"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.127350 12353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:44:19.130242 12368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.131428 12353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 13:44:19.131539 12353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "bead919308d14384af5b62af8cd67cc6"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "e5906136e9d3242679ca20470b15b3aa"
server_key_iv: "d5e7f24392f43c84549e55fa962922ab"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.131696 12353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:19.160142 12353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:19.163678 12353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:19.163892 12353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:19.164667 12353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:19.165997 12353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:19.166095 12353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.166234 12353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:19.166303 12353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.197824 12353 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:38539
I20260927 13:44:19.197901 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:38539 every 8 connection(s)
I20260927 13:44:19.199024 12353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:44:19.206393 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 12353
I20260927 13:44:19.206619 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:44:19.207273 1975 external_mini_cluster.cc:1428] Setting key cfba4b1cc3f90e0c53e00a6d213f9980
Sep 27 13:44:19 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:44:19.217478 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:19.201334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:41185 (local address 127.1.237.254:39295)
0927 13:44:19.201819 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:19.201823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:19.202568 (+ 745us) server_negotiation.cc:407] Connection header received
0927 13:44:19.204302 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:19.204307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:19.204391 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:19.204505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:19.205830 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.206621 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.208248 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.208626 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.208753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:19.212001 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:19.212006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:19.212009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:19.212035 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:19.214484 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:19.215140 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:19.215146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:19.215151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:19.215294 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:19.215692 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:19.215697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:19.215700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:19.216060 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:19.216215 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:19.216918 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 13:44:19.217153 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":281,"threads_started":1}
I20260927 13:44:19.218258 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:19.201731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:39295 (local address 127.1.237.193:41185)
0927 13:44:19.202404 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:19.202446 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:19.204054 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:19.204724 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:19.204731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:19.205162 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:19.205646 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:19.205662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.206793 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.206796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:19.208094 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:19.208104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.208832 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.208840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:19.209061 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:19.209632 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:19.209652 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:19.211809 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:19.214625 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:19.214633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:19.214647 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:19.215018 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:19.215400 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:19.215406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:19.215409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:19.215580 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:19.216187 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:19.216195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:19.216598 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 13:44:19.216834 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 13:44:19.217278 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":294,"threads_started":1}
I20260927 13:44:19.219681 12481 heartbeater.cc:347] Connected to a master server at 127.1.237.254:39295
I20260927 13:44:19.219941 12481 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:19.220573 12481 heartbeater.cc:511] Master 127.1.237.254:39295 requested a full tablet report, sending...
I20260927 13:44:19.222549 12292 ts_manager.cc:194] Registered new tserver with Master: bead919308d14384af5b62af8cd67cc6 (127.1.237.193:38539)
I20260927 13:44:19.224639 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:41185
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:44:19.252638 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:44:19.558969 12487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:19.559278 12487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:19.559445 12487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:19.566362 12487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:19.566450 12487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:19.566555 12487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:44:19.575706 12487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:19.576745 12487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:19.577962 12487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:19.590217 12492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:19.590289 12493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:19.590260 12495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:19.590966 12487 server_base.cc:1034] running on GCE node
I20260927 13:44:19.591603 12487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:19.592484 12487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:19.593703 12487 hybrid_clock.cc:648] HybridClock initialized: now 1790516659593672 us; error 51 us; skew 500 ppm
I20260927 13:44:19.594002 12487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:44:19 dist-test-slave-7g3l krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:44:19.596920 12487 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:44:19.598260 12487 webserver.cc:460] Webserver started at http://127.1.237.194:32891/ using document root <none> and password file <none>
I20260927 13:44:19.598857 12487 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:19.598994 12487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:19.599400 12487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:19.602005 12487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "fe8467b506ef430bad6bd55017ef8a50"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "ec5ecaab3b4cea9ec165681b6cf3ec63"
server_key_iv: "fb95c9674d7deef3a0fe19ca7096d1e6"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.602648 12487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "fe8467b506ef430bad6bd55017ef8a50"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "ec5ecaab3b4cea9ec165681b6cf3ec63"
server_key_iv: "fb95c9674d7deef3a0fe19ca7096d1e6"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.607329 12487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 13:44:19.610338 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.611558 12487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:44:19.611725 12487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "fe8467b506ef430bad6bd55017ef8a50"
format_stamp: "Formatted at 2026-09-27 13:44:19 on dist-test-slave-7g3l"
server_key: "ec5ecaab3b4cea9ec165681b6cf3ec63"
server_key_iv: "fb95c9674d7deef3a0fe19ca7096d1e6"
server_key_version: "encryptionkey@0"
I20260927 13:44:19.611927 12487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:19.631693 12487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:19.635444 12487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:19.635700 12487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:19.636559 12487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:19.637776 12487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:19.637880 12487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.638025 12487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:19.638096 12487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:19.670156 12487 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:40313
I20260927 13:44:19.670198 12614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:40313 every 8 connection(s)
I20260927 13:44:19.671446 12487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:44:19.681185 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 12487
I20260927 13:44:19.681381 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:44:19.681802 1975 external_mini_cluster.cc:1428] Setting key c674e0811166c0b4eb4f423146d9c649
Sep 27 13:44:19 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516659, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:44:19.689922 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:19.673897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:41155 (local address 127.1.237.254:39295)
0927 13:44:19.674059 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:19.674063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:19.675217 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 13:44:19.676960 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:19.676965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:19.677050 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:19.677160 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:19.678525 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.679284 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.680683 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.681077 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.681195 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:19.684688 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:19.684701 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:19.684704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:19.684731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:19.687077 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:19.687765 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:19.687771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:19.687774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:19.687894 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:19.688295 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:19.688299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:19.688302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:19.688611 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:19.688762 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:19.689389 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 13:44:19.689592 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:44:19.690804 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:19.674414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:39295 (local address 127.1.237.194:41155)
0927 13:44:19.675061 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:19.675095 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:19.676695 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:19.677379 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:19.677387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:19.677840 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:19.678348 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:19.678364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.679430 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.679434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:19.680544 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:19.680555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:19.681290 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:19.681297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:19.681557 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:19.682169 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:19.682193 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:19.684506 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:19.687230 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:19.687239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:19.687252 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:19.687651 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:19.688007 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:19.688013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:19.688016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:19.688181 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:19.688729 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:19.688738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:19.689056 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 13:44:19.689315 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 13:44:19.689796 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":248,"threads_started":1}
I20260927 13:44:19.692198 12615 heartbeater.cc:347] Connected to a master server at 127.1.237.254:39295
I20260927 13:44:19.692478 12615 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:19.693112 12615 heartbeater.cc:511] Master 127.1.237.254:39295 requested a full tablet report, sending...
I20260927 13:44:19.694641 12292 ts_manager.cc:194] Registered new tserver with Master: fe8467b506ef430bad6bd55017ef8a50 (127.1.237.194:40313)
I20260927 13:44:19.695427 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:41155
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:44:19.727816 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:44:20.030035 12620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:44:20.030282 12620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:44:20.030369 12620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:44:20.037581 12620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:44:20.037672 12620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:44:20.037786 12620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:44:20.046845 12620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:44:20.047915 12620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:44:20.049044 12620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:44:20.061293 12625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:20.061267 12626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:44:20.061370 12628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:44:20.062108 12620 server_base.cc:1034] running on GCE node
I20260927 13:44:20.062620 12620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:44:20.063474 12620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:44:20.064689 12620 hybrid_clock.cc:648] HybridClock initialized: now 1790516660064660 us; error 42 us; skew 500 ppm
I20260927 13:44:20.064989 12620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:44:20 dist-test-slave-7g3l krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516660, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:44:20.067840 12620 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:44:20.069151 12620 webserver.cc:460] Webserver started at http://127.1.237.195:36931/ using document root <none> and password file <none>
I20260927 13:44:20.069761 12620 fs_manager.cc:359] Metadata directory not provided
I20260927 13:44:20.069908 12620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:44:20.070187 12620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:44:20.072763 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "3a7dff939dee4f33b8dc308b0af11b0d"
format_stamp: "Formatted at 2026-09-27 13:44:20 on dist-test-slave-7g3l"
server_key: "fb242160aea4e55486cf85f022cb0f50"
server_key_iv: "867cbd38422ee6e67b27956c36155ac7"
server_key_version: "encryptionkey@0"
I20260927 13:44:20.073536 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "3a7dff939dee4f33b8dc308b0af11b0d"
format_stamp: "Formatted at 2026-09-27 13:44:20 on dist-test-slave-7g3l"
server_key: "fb242160aea4e55486cf85f022cb0f50"
server_key_iv: "867cbd38422ee6e67b27956c36155ac7"
server_key_version: "encryptionkey@0"
I20260927 13:44:20.078164 12620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 13:44:20.081368 12635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:20.082489 12620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:44:20.082646 12620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "3a7dff939dee4f33b8dc308b0af11b0d"
format_stamp: "Formatted at 2026-09-27 13:44:20 on dist-test-slave-7g3l"
server_key: "fb242160aea4e55486cf85f022cb0f50"
server_key_iv: "867cbd38422ee6e67b27956c36155ac7"
server_key_version: "encryptionkey@0"
I20260927 13:44:20.082831 12620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:44:20.099116 12620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:44:20.102689 12620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:44:20.102960 12620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:44:20.103821 12620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:44:20.105000 12620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:44:20.105069 12620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:20.105232 12620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:44:20.105281 12620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:44:20.137146 12620 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34759
I20260927 13:44:20.137207 12747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34759 every 8 connection(s)
I20260927 13:44:20.138238 12620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:44:20.144893 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 12620
I20260927 13:44:20.145089 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:44:20.145623 1975 external_mini_cluster.cc:1428] Setting key d10e0b4a848ecf7eace5afda08e1257a
Sep 27 13:44:20 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516660, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:44:20.156082 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.140551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:46851 (local address 127.1.237.254:39295)
0927 13:44:20.140732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.140737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.141800 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 13:44:20.143519 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.143524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.143584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.143665 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:20.144947 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.146115 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.147823 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.148343 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.148476 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.151468 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:20.151473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:20.151476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:20.151504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:20.153779 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.154334 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.154338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.154340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.154437 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.154752 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.154755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.154757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.154979 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:20.155075 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.155681 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.155846 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:44:20.156822 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.141039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:39295 (local address 127.1.237.195:46851)
0927 13:44:20.141657 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.141690 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.143298 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.143871 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.143878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.144318 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:44:20.144792 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.144807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.146272 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.146278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.147686 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.147697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.148539 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.148544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.148724 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.149319 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:44:20.149344 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:44:20.151308 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:44:20.153882 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:20.153888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:20.153896 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:20.154238 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:20.154518 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:44:20.154521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:44:20.154523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:44:20.154664 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:44:20.155072 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:44:20.155076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:44:20.155380 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.155590 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.155902 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":251,"threads_started":1}
I20260927 13:44:20.158056 12748 heartbeater.cc:347] Connected to a master server at 127.1.237.254:39295
I20260927 13:44:20.158311 12748 heartbeater.cc:464] Registering TS with master...
I20260927 13:44:20.159055 12748 heartbeater.cc:511] Master 127.1.237.254:39295 requested a full tablet report, sending...
I20260927 13:44:20.160429 12292 ts_manager.cc:194] Registered new tserver with Master: 3a7dff939dee4f33b8dc308b0af11b0d (127.1.237.195:34759)
I20260927 13:44:20.161103 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:46851
I20260927 13:44:20.172180 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:44:20.187858 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.176271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.1.237.254:39295)
0927 13:44:20.176411 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.176415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.176868 (+ 453us) server_negotiation.cc:407] Connection header received
0927 13:44:20.176982 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.176986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.177040 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.177124 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:20.178390 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.179121 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.180275 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.180645 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.180738 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.182657 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:20.182661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:20.182664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:20.182689 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:20.185180 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.185846 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.185853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.185856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.185980 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.186384 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.186390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.186393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.186685 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:20.186939 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.187353 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.187571 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:44:20.191625 12292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:44:20.198326 12292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:44:20.227396 12481 heartbeater.cc:503] Master 127.1.237.254:39295 was elected leader, sending a full tablet report...
I20260927 13:44:20.229779 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.210371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34759 (local address 127.0.0.1:38214)
0927 13:44:20.221454 (+ 11083us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.221469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.221576 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.222318 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.222321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.222356 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.223302 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.223314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.224968 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.224973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.226936 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.226947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.228927 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.228934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.229111 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.229166 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.229298 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.229475 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10981,"mutex_wait_us":321,"spinlock_wait_cycles":14080,"thread_start_us":10060,"threads_started":1}
I20260927 13:44:20.229978 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.209623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38539 (local address 127.0.0.1:53900)
0927 13:44:20.220766 (+ 11143us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.220802 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.220958 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.222300 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.222308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.222351 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.222889 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.222900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.224805 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.224810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.226464 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.226475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.229108 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.229114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.229307 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.229337 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.229420 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.229539 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10941,"mutex_wait_us":123,"spinlock_wait_cycles":384,"thread_start_us":11577,"threads_started":1}
I20260927 13:44:20.230609 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.211000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.1.237.195:34759)
0927 13:44:20.219802 (+ 8802us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.219814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.221678 (+ 1864us) server_negotiation.cc:407] Connection header received
0927 13:44:20.221759 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.221765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.221947 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.222086 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.223529 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.224830 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.227076 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.228403 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.228523 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.228768 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.230190 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.230332 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8086,"thread_start_us":7908,"threads_started":1}
I20260927 13:44:20.230666 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.209971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.1.237.193:38539)
0927 13:44:20.221457 (+ 11486us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.221467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.221498 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:44:20.221594 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.221603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.221847 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.222051 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.223061 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.224668 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.226664 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.228617 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.228749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.228968 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.230171 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.230335 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11336,"thread_start_us":11174,"threads_started":1}
I20260927 13:44:20.231590 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.221581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:40313 (local address 127.0.0.1:53120)
0927 13:44:20.223483 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.223499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.223599 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.224366 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.224371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.224393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.224698 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.224710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.226566 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.226573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.228835 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.228848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.230629 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.230636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.231052 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.231082 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.231199 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.231313 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"thread_start_us":1645,"threads_started":1}
I20260927 13:44:20.231698 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.221810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.1.237.194:40313)
0927 13:44:20.222783 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.222792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.223699 (+ 907us) server_negotiation.cc:407] Connection header received
0927 13:44:20.223781 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.223789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.223993 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.224159 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.224869 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.226435 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.229012 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.230422 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.230556 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.230735 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.231238 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.231387 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":685,"threads_started":1}
I20260927 13:44:20.233959 12683 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 (DEFAULT_TABLE table=test-table [id=52650c52bc734513800a3af2d24b8e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:44:20.234150 12416 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 (DEFAULT_TABLE table=test-table [id=52650c52bc734513800a3af2d24b8e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:44:20.235329 12683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6158caaf69407280f1258f8cb2fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:20.235451 12416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6158caaf69407280f1258f8cb2fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:20.234995 12550 tablet_service.cc:1467] Processing CreateTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 (DEFAULT_TABLE table=test-table [id=52650c52bc734513800a3af2d24b8e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:44:20.236245 12550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e6158caaf69407280f1258f8cb2fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:44:20.245636 12766 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Bootstrap starting.
I20260927 13:44:20.248601 12766 tablet_bootstrap.cc:654] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:20.248752 12767 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Bootstrap starting.
I20260927 13:44:20.250200 12768 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Bootstrap starting.
I20260927 13:44:20.250909 12766 log.cc:824] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:20.251915 12767 tablet_bootstrap.cc:654] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:20.253084 12767 log.cc:824] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:20.253168 12768 tablet_bootstrap.cc:654] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Neither blocks nor log segments found. Creating new log.
I20260927 13:44:20.253934 12766 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: No bootstrap required, opened a new log
I20260927 13:44:20.254168 12766 ts_tablet_manager.cc:1397] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 13:44:20.254252 12768 log.cc:824] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Log is configured to *not* fsync() on all Append() calls
I20260927 13:44:20.255828 12767 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: No bootstrap required, opened a new log
I20260927 13:44:20.256062 12767 ts_tablet_manager.cc:1397] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 13:44:20.256995 12768 tablet_bootstrap.cc:492] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: No bootstrap required, opened a new log
I20260927 13:44:20.257225 12768 ts_tablet_manager.cc:1397] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 13:44:20.262674 12767 raft_consensus.cc:348] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.262977 12767 raft_consensus.cc:374] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:20.263059 12767 raft_consensus.cc:729] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bead919308d14384af5b62af8cd67cc6, State: Initialized, Role: FOLLOWER
I20260927 13:44:20.263572 12767 consensus_queue.cc:260] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.264227 12766 raft_consensus.cc:348] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.264576 12766 raft_consensus.cc:374] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:20.264653 12766 raft_consensus.cc:729] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7dff939dee4f33b8dc308b0af11b0d, State: Initialized, Role: FOLLOWER
I20260927 13:44:20.265045 12767 ts_tablet_manager.cc:1428] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260927 13:44:20.265209 12766 consensus_queue.cc:260] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.266474 12748 heartbeater.cc:503] Master 127.1.237.254:39295 was elected leader, sending a full tablet report...
I20260927 13:44:20.267069 12766 ts_tablet_manager.cc:1428] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 13:44:20.266960 12768 raft_consensus.cc:348] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.267362 12768 raft_consensus.cc:374] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:44:20.267464 12768 raft_consensus.cc:729] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8467b506ef430bad6bd55017ef8a50, State: Initialized, Role: FOLLOWER
I20260927 13:44:20.268044 12768 consensus_queue.cc:260] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.269049 12615 heartbeater.cc:503] Master 127.1.237.254:39295 was elected leader, sending a full tablet report...
I20260927 13:44:20.269642 12768 ts_tablet_manager.cc:1428] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260927 13:44:20.389744 12749 tablet.cc:2369] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:44:20.423337 12616 tablet.cc:2369] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:44:20.451135 12482 tablet.cc:2369] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:44:20.682960 12774 raft_consensus.cc:482] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:44:20.683243 12774 raft_consensus.cc:504] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.684841 12774 leader_election.cc:288] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bead919308d14384af5b62af8cd67cc6 (127.1.237.193:38539), 3a7dff939dee4f33b8dc308b0af11b0d (127.1.237.195:34759)
I20260927 13:44:20.689612 12773 raft_consensus.cc:482] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:44:20.689847 12773 raft_consensus.cc:504] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.691495 12773 leader_election.cc:288] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bead919308d14384af5b62af8cd67cc6 (127.1.237.193:38539), fe8467b506ef430bad6bd55017ef8a50 (127.1.237.194:40313)
I20260927 13:44:20.692688 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.685252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:35427 (local address 127.1.237.193:38539)
0927 13:44:20.685473 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.685482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.686193 (+ 711us) server_negotiation.cc:407] Connection header received
0927 13:44:20.686364 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.686371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.686468 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.686556 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.687330 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.688334 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.690467 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.691722 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.691881 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.691953 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.692303 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.692450 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:44:20.692821 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.685251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38539 (local address 127.1.237.194:35427)
0927 13:44:20.686094 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.686110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.686218 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.686583 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.686587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.686609 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.687143 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.687190 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.688458 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.688464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.690276 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.690289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.691931 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.691940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.692115 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.692152 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.692268 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.692420 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":36,"thread_start_us":466,"threads_started":1}
I20260927 13:44:20.693923 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.685253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:52693 (local address 127.1.237.195:34759)
0927 13:44:20.685538 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.685544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.686363 (+ 819us) server_negotiation.cc:407] Connection header received
0927 13:44:20.686546 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.686551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.686633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.686720 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.687444 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.688447 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.690391 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.691722 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.691879 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.691953 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.693541 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.693671 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260927 13:44:20.693799 12436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "fe8467b506ef430bad6bd55017ef8a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bead919308d14384af5b62af8cd67cc6" is_pre_election: true
I20260927 13:44:20.694063 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.685272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34759 (local address 127.1.237.194:52693)
0927 13:44:20.686224 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.686240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.686399 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.686740 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.686744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.686767 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.687281 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.687292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.688571 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.688577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.690239 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.690250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.693094 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.693103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.693348 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.693400 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.693504 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.693670 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":480,"threads_started":1}
I20260927 13:44:20.694331 12436 raft_consensus.cc:2454] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8467b506ef430bad6bd55017ef8a50 in term 0.
I20260927 13:44:20.695020 12504 leader_election.cc:302] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bead919308d14384af5b62af8cd67cc6, fe8467b506ef430bad6bd55017ef8a50; no voters:
I20260927 13:44:20.695461 12774 raft_consensus.cc:2781] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:44:20.695600 12774 raft_consensus.cc:482] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:44:20.695688 12774 raft_consensus.cc:3037] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:20.695550 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "fe8467b506ef430bad6bd55017ef8a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" is_pre_election: true
I20260927 13:44:20.696005 12703 raft_consensus.cc:2454] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8467b506ef430bad6bd55017ef8a50 in term 0.
I20260927 13:44:20.697876 12774 raft_consensus.cc:504] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.698407 12774 leader_election.cc:288] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [CANDIDATE]: Term 1 election: Requested vote from peers bead919308d14384af5b62af8cd67cc6 (127.1.237.193:38539), 3a7dff939dee4f33b8dc308b0af11b0d (127.1.237.195:34759)
I20260927 13:44:20.698721 12436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "fe8467b506ef430bad6bd55017ef8a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bead919308d14384af5b62af8cd67cc6"
I20260927 13:44:20.698913 12436 raft_consensus.cc:3037] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:20.699012 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "fe8467b506ef430bad6bd55017ef8a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a7dff939dee4f33b8dc308b0af11b0d"
I20260927 13:44:20.699323 12703 raft_consensus.cc:3037] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:44:20.702471 12436 raft_consensus.cc:2454] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8467b506ef430bad6bd55017ef8a50 in term 1.
I20260927 13:44:20.702839 12504 leader_election.cc:302] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bead919308d14384af5b62af8cd67cc6, fe8467b506ef430bad6bd55017ef8a50; no voters:
I20260927 13:44:20.703102 12774 raft_consensus.cc:2781] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:44:20.703590 12703 raft_consensus.cc:2454] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8467b506ef430bad6bd55017ef8a50 in term 1.
I20260927 13:44:20.705715 12774 raft_consensus.cc:686] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 1 LEADER]: Becoming Leader. State: Replica: fe8467b506ef430bad6bd55017ef8a50, State: Running, Role: LEADER
I20260927 13:44:20.706422 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.692172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:60059 (local address 127.1.237.193:38539)
0927 13:44:20.692970 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.692974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.692991 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:44:20.695190 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.695196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.695261 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.695355 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.699563 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.700707 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.703424 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.705111 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.705280 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.705383 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.706037 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.706152 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":3358,"threads_started":1}
I20260927 13:44:20.706248 12774 consensus_queue.cc:237] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } }
I20260927 13:44:20.706446 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.691750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38539 (local address 127.1.237.195:60059)
0927 13:44:20.692327 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.692344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.692443 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.698702 (+ 6259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.698710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.698739 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.699332 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.699348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.700913 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.700918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.703255 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.703269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.705583 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.705592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.705869 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.705901 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.705998 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.706119 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":645,"threads_started":1}
I20260927 13:44:20.707417 12436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bead919308d14384af5b62af8cd67cc6" is_pre_election: true
I20260927 13:44:20.707700 12436 raft_consensus.cc:2379] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a7dff939dee4f33b8dc308b0af11b0d in current term 1: Already voted for candidate fe8467b506ef430bad6bd55017ef8a50 in this term.
I20260927 13:44:20.711293 12292 catalog_manager.cc:5495] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8467b506ef430bad6bd55017ef8a50 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "fe8467b506ef430bad6bd55017ef8a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8467b506ef430bad6bd55017ef8a50" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 } health_report { overall_health: UNKNOWN } } }
I20260927 13:44:20.713517 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.692788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:40313 (local address 127.1.237.195:52093)
0927 13:44:20.705431 (+ 12643us) negotiation.cc:107] Waiting for socket to connect
0927 13:44:20.705472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:44:20.706113 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:44:20.706922 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:44:20.706925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:44:20.706945 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:44:20.708449 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.708465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.709969 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.709974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.711793 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:44:20.711808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.712989 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.712993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:44:20.713112 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.713133 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:44:20.713195 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 13:44:20.713275 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12546,"thread_start_us":12770,"threads_started":1}
I20260927 13:44:20.713608 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.693348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:52093 (local address 127.1.237.194:40313)
0927 13:44:20.694821 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.694826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.705550 (+ 10724us) server_negotiation.cc:407] Connection header received
0927 13:44:20.706518 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.706523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.706588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.706648 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:44:20.708659 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.709829 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.712003 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.712811 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.712960 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.713020 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.713263 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.713372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1349}
I20260927 13:44:20.714357 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e6158caaf69407280f1258f8cb2fa41" candidate_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8467b506ef430bad6bd55017ef8a50" is_pre_election: true
I20260927 13:44:20.715338 12637 leader_election.cc:302] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a7dff939dee4f33b8dc308b0af11b0d; no voters: bead919308d14384af5b62af8cd67cc6, fe8467b506ef430bad6bd55017ef8a50
I20260927 13:44:20.715725 12773 raft_consensus.cc:2726] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 13:44:20.781710 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.776531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.1.237.194:40313)
0927 13:44:20.776685 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.776690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.777145 (+ 455us) server_negotiation.cc:407] Connection header received
0927 13:44:20.777311 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.777315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.777394 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.777510 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:44:20.777926 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.778679 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.779982 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.780341 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.780426 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.780832 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:44:20.781286 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:44:20.781383 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.781464 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.781518 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:44:20.789052 12436 raft_consensus.cc:1261] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 1 FOLLOWER]: Refusing update from remote peer fe8467b506ef430bad6bd55017ef8a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:44:20.789096 12703 raft_consensus.cc:1261] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 1 FOLLOWER]: Refusing update from remote peer fe8467b506ef430bad6bd55017ef8a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:44:20.789831 12780 consensus_queue.cc:1035] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bead919308d14384af5b62af8cd67cc6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:44:20.790179 12774 consensus_queue.cc:1035] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7dff939dee4f33b8dc308b0af11b0d" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:44:20.803560 12791 mvcc.cc:205] Tried to move back new op lower bound from 7333956242582573056 to 7333956242257469440. Current Snapshot: MvccSnapshot[applied={T|T < 7333956242582573056}]
I20260927 13:44:20.804672 12789 mvcc.cc:205] Tried to move back new op lower bound from 7333956242582573056 to 7333956242257469440. Current Snapshot: MvccSnapshot[applied={T|T < 7333956242582573056}]
I20260927 13:44:20.820102 12292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:44:20.820335 12292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:44:20.823369 12292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c832e3952eea4f54bf5f5165ebcbc0a2: Sending DeleteTablet for 3 replicas of tablet 2e6158caaf69407280f1258f8cb2fa41
I20260927 13:44:20.824353 12416 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:44:20 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53900
I20260927 13:44:20.824429 12683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:44:20 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260927 13:44:20.824462 12550 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e6158caaf69407280f1258f8cb2fa41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:44:20 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53120
I20260927 13:44:20.826646 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 12353
I20260927 13:44:20.860046 12799 tablet_replica.cc:330] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: stopping tablet replica
I20260927 13:44:20.860250 12798 tablet_replica.cc:330] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: stopping tablet replica
I20260927 13:44:20.860500 12799 raft_consensus.cc:2229] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:44:20.860628 12798 raft_consensus.cc:2229] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:44:20.861012 12799 raft_consensus.cc:2258] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:44:20.861095 12798 raft_consensus.cc:2258] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:44:20.862838 12799 ts_tablet_manager.cc:1905] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:44:20.863076 12798 ts_tablet_manager.cc:1905] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:44:20 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:44:20.869901 12799 ts_tablet_manager.cc:1918] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:44:20.869984 12798 ts_tablet_manager.cc:1918] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:44:20.870046 12799 log.cc:1196] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/2e6158caaf69407280f1258f8cb2fa41
I20260927 13:44:20.870141 12798 log.cc:1196] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/2e6158caaf69407280f1258f8cb2fa41
I20260927 13:44:20.870584 12799 ts_tablet_manager.cc:1939] T 2e6158caaf69407280f1258f8cb2fa41 P 3a7dff939dee4f33b8dc308b0af11b0d: Deleting consensus metadata
I20260927 13:44:20.870635 12798 ts_tablet_manager.cc:1939] T 2e6158caaf69407280f1258f8cb2fa41 P bead919308d14384af5b62af8cd67cc6: Deleting consensus metadata
I20260927 13:44:20.861266 12797 tablet_replica.cc:330] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: stopping tablet replica
I20260927 13:44:20.871592 12797 raft_consensus.cc:2229] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:44:20.872228 12797 raft_consensus.cc:2258] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:44:20.872457 12277 catalog_manager.cc:4841] TS bead919308d14384af5b62af8cd67cc6 (127.1.237.193:38539): tablet 2e6158caaf69407280f1258f8cb2fa41 (table test-table [id=52650c52bc734513800a3af2d24b8e72]) successfully deleted
I20260927 13:44:20.873310 12278 catalog_manager.cc:4841] TS 3a7dff939dee4f33b8dc308b0af11b0d (127.1.237.195:34759): tablet 2e6158caaf69407280f1258f8cb2fa41 (table test-table [id=52650c52bc734513800a3af2d24b8e72]) successfully deleted
I20260927 13:44:20.873587 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:20.828200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.1.237.193:38539)
0927 13:44:20.828404 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:20.828413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:20.851757 (+ 23344us) server_negotiation.cc:407] Connection header received
0927 13:44:20.852243 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:20.852252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:20.852362 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:20.852446 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:20.854506 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.856061 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.859597 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:20.860227 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:20.860338 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:20.865591 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:20.865603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:20.865620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:20.865655 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:20.869482 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.870366 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.870376 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.870379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.870550 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:20.871028 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:20.871033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:20.871035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:20.871498 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:20.871810 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:20.872997 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0927 13:44:20.873249 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:44:20.874153 12797 ts_tablet_manager.cc:1905] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:44:20.874704 12476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:20.881103 12797 ts_tablet_manager.cc:1918] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:44:20.881258 12797 log.cc:1196] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/2e6158caaf69407280f1258f8cb2fa41
I20260927 13:44:20.881700 12797 ts_tablet_manager.cc:1939] T 2e6158caaf69407280f1258f8cb2fa41 P fe8467b506ef430bad6bd55017ef8a50: Deleting consensus metadata
I20260927 13:44:20.882900 12277 catalog_manager.cc:4841] TS fe8467b506ef430bad6bd55017ef8a50 (127.1.237.194:40313): tablet 2e6158caaf69407280f1258f8cb2fa41 (table test-table [id=52650c52bc734513800a3af2d24b8e72]) successfully deleted
I20260927 13:44:20.984700 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 12353
I20260927 13:44:21.001652 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 12487
Sep 27 13:44:21 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:44:21.016522 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:21.002438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.1.237.194:40313)
0927 13:44:21.002643 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:21.002649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:21.002674 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:44:21.002877 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:21.002882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:21.002962 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:21.003063 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:21.004226 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:21.004989 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:21.006195 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:21.006612 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:21.006769 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:21.010544 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:21.010559 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:21.010575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:21.010611 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:21.013672 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:21.014411 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:21.014421 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:21.014423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:21.014542 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:21.014934 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:21.014940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:21.014944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:21.015390 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:21.015698 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:21.015968 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:44:21.016197 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:44:21.017270 12610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:21.125573 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 12487
I20260927 13:44:21.145805 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 12620
Sep 27 13:44:21 dist-test-slave-7g3l krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:44:21.161667 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:44:21.146573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.1.237.195:34759)
0927 13:44:21.146755 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:44:21.146767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:44:21.146786 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:44:21.146847 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:44:21.146852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:44:21.146930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:44:21.147025 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:44:21.148192 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:21.149338 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:21.150493 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:44:21.151092 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:44:21.151308 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:44:21.155366 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:44:21.155379 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:44:21.155396 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:44:21.155436 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:44:21.159013 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:21.159635 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:21.159643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:21.159646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:21.159803 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:44:21.160135 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:44:21.160139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:44:21.160142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:44:21.160580 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:44:21.160861 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 13:44:21.161116 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 13:44:21.161319 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:44:21.162386 12743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:21.249835 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 12620
I20260927 13:44:21.268093 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 12260
I20260927 13:44:21.269172 12322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:44:21.370478 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 12260
2026-09-27T13:44:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4388 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:44:21 dist-test-slave-7g3l krb5kdc[12807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:44:21 dist-test-slave-7g3l krb5kdc[12807](info): set up 2 sockets
Sep 27 13:44:21 dist-test-slave-7g3l krb5kdc[12807](info): commencing operation
krb5kdc: starting...
W20260927 13:44:22.441835 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:44:22 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:44:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangeradmin_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangeradmin_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangerlookup_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangerlookup_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
I20260927 13:44:22.720293 1975 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskijlVCN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/postgres -l logfile start
2026-09-27 13:44:24.968 UTC [12837] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:44:24.968 UTC [12837] LOG: listening on IPv4 address "127.1.237.212", port 41221
2026-09-27 13:44:24.971 UTC [12837] LOG: listening on Unix socket "/tmp/.s.PGSQL.41221"
2026-09-27 13:44:24.992 UTC [12840] LOG: database system was shut down at 2026-09-27 13:44:23 UTC
2026-09-27 13:44:24.995 UTC [12837] LOG: database system is ready to accept connections
I20260927 13:44:25.947594 1975 mini_postgres.cc:96] Postgres bound to 41221
2026-09-27 13:44:25.963 UTC [12848] FATAL: database "slave" does not exist
127.1.237.212:41221 - accepting connections
I20260927 13:44:25.964334 1975 mini_ranger.cc:161] Starting Ranger...
I20260927 13:44:25.987496 1975 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:44:26.395117 1975 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:44:26.395319 1975 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin
2026-09-27 13:44:26,676 [I] DB FLAVOR :POSTGRES
2026-09-27 13:44:26,677 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:44:26,677 [I] Checking connection..
2026-09-27 13:44:26,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:44:26,926 [I] Checking connection passed.
2026-09-27 13:44:26,927 [I] --------- Verifying version history table ---------
2026-09-27 13:44:26,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:44:27,144 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:44:27,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:44:27,386 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:44:27,387 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:44:27,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:44:27.617 UTC [12912] WARNING: there is no transaction in progress
2026-09-27 13:44:27,623 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:44:27,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:44:27,874 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:44:27,875 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:44:27,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:44:28,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:44:28,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7g3l.c.gcp-upstream.internal','N') ;"
2026-09-27 13:44:28,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:44:28,792 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:44:28,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:44:29,030 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:44:29,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:44:29,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:44:29,481 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:44:29,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:44:30.333 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.347 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.361 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.374 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.388 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.402 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.461 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.467 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.481 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.490 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.499 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.511 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.519 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.525 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.531 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:30.537 UTC [13020] WARNING: there is no transaction in progress
2026-09-27 13:44:31,082 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:44:31,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:44:31,314 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:44:31,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:44:31,540 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:44:31,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:44:31,772 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:44:31,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:44:32,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:44:32,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7g3l.c.gcp-upstream.internal';"
2026-09-27 13:44:32,498 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:44:32,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:44:32,748 [I] DB_PATCHES have already been applied
I20260927 13:44:32.754674 1975 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskijlVCN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:44:32.754758 1975 mini_ranger.cc:193] Using host: 127.1.237.212
I20260927 13:44:32.768383 1975 mini_ranger.cc:227] Ranger admin URL: http://127.1.237.212:45855
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:44:33.877884 1975 mini_ranger.cc:160] Time spent starting Ranger: real 7.914s user 0.004s sys 0.029s
Sep 27, 2026 1:44:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:44:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 1:44:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:44:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:44:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:44:53.175074 1975 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangerkms_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/rangerkms_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.237.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.237.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:44:53 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516693, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:44:53.356141 1975 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:44:53.376435 1975 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:44:54.144462 1975 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:44:54.144637 1975 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms
I20260927 13:44:54.144773 1975 mini_ranger_kms.cc:227] Using postgres at 127.1.237.212:41221
2026-09-27 13:44:54,537 [I] DB FLAVOR :POSTGRES
2026-09-27 13:44:54,537 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:44:54,538 [I] Checking connection
2026-09-27 13:44:54,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:44:54,758 [I] connection success
2026-09-27 13:44:54,758 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:44:54,758 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:44:54,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:44:54,982 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:44:54,982 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:44:54,982 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:44:54,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:41221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:44:55,246 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:44:55.688202 1975 mini_ranger_kms.cc:310] Created kms service
I20260927 13:44:55.855682 1975 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:44:55.899628 1975 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:44:55 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516693, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:44:56.288333 1975 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:44:56.288640 1975 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms:/tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:44:56.288691 1975 mini_ranger_kms.cc:240] Using host: 127.1.237.212
I20260927 13:44:56.311393 1975 mini_ranger_kms.cc:276] Ranger KMS PID: 13200
I20260927 13:44:56.311553 1975 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.237.212:58323
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:44:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 13:44:58 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790516698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:44:58.571729 1975 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.216s user 0.112s sys 0.300s
I20260927 13:44:58.571973 1975 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:44:58.572185 1975 mini_ranger_kms.cc:401] 127.1.237.212:58323/kms/v1/keys
Sep 27 13:44:58 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
Sep 27 13:44:58 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516698, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
Sep 27 13:45:01 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:45:01.124224 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:44345
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:44345
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:58323/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:45:01.454126 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:01.454483 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:01.454643 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:01.462788 13251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:01.462930 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:01.463034 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:01.463143 13251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:01.463265 13251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:01.473171 13251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.237.212:58323/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:44345
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:01.474459 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:01.475849 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:01.489223 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:01.489069 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:01.489641 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:01.491420 13251 server_base.cc:1034] running on GCE node
I20260927 13:45:01.492393 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:01.494216 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:01.495455 13251 hybrid_clock.cc:648] HybridClock initialized: now 1790516701495408 us; error 40 us; skew 500 ppm
I20260927 13:45:01.495877 13251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:01 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:01.503309 13251 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:45:01.505409 13251 webserver.cc:460] Webserver started at http://127.1.237.254:35959/ using document root <none> and password file <none>
I20260927 13:45:01.506114 13251 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:01.506235 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:01.506573 13251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:01 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:01.566726 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "125cfb0f8c7f4200a4b02581f451b99e"
format_stamp: "Formatted at 2026-09-27 13:45:01 on dist-test-slave-7g3l"
server_key: "798c811c63a9a0aae1c86aaa007f8bea"
server_key_iv: "b0510bf47f28b4f53b3b3919b65e2619"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:01.567763 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "125cfb0f8c7f4200a4b02581f451b99e"
format_stamp: "Formatted at 2026-09-27 13:45:01 on dist-test-slave-7g3l"
server_key: "798c811c63a9a0aae1c86aaa007f8bea"
server_key_iv: "b0510bf47f28b4f53b3b3919b65e2619"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:01.574518 13251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 13:45:01.594548 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:01.596517 13251 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:45:01.596716 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "125cfb0f8c7f4200a4b02581f451b99e"
format_stamp: "Formatted at 2026-09-27 13:45:01 on dist-test-slave-7g3l"
server_key: "798c811c63a9a0aae1c86aaa007f8bea"
server_key_iv: "b0510bf47f28b4f53b3b3919b65e2619"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:01.596944 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:01.638278 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:01.643080 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:01.643409 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:01.663796 13251 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:44345
I20260927 13:45:01.663829 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:44345 every 8 connection(s)
I20260927 13:45:01.665241 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:45:01.669018 13318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:01.669965 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 13251
I20260927 13:45:01.670181 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:45:01.670845 1975 external_mini_cluster.cc:1428] Setting key 53a6ab3649838a80cbe240802a55a1c0
I20260927 13:45:01.681495 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: Bootstrap starting.
I20260927 13:45:01.685616 13318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:01.686828 13318 log.cc:824] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: Log is configured to *not* fsync() on all Append() calls
Sep 27 13:45:01 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:01.689684 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: No bootstrap required, opened a new log
I20260927 13:45:01.697162 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:01.672790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.1.237.254:44345)
0927 13:45:01.675871 (+ 3081us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:01.675889 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:01.676234 (+ 345us) server_negotiation.cc:407] Connection header received
0927 13:45:01.678094 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:01.678115 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:01.678478 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:01.678893 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:01.680212 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:01.681614 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:01.683830 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:01.684499 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:01.684648 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:01.689373 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:01.689381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:01.689396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:01.689434 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:01.692707 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:01.693439 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:01.693445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:01.693452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:01.693607 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:01.693994 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:01.694014 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:01.694016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:01.694547 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:01.694888 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:01.695312 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 13:45:01.695708 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2800,"thread_start_us":2576,"threads_started":1}
I20260927 13:45:01.697629 13318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } }
I20260927 13:45:01.697997 13318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:01.698108 13318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125cfb0f8c7f4200a4b02581f451b99e, State: Initialized, Role: FOLLOWER
I20260927 13:45:01.698714 13318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } }
I20260927 13:45:01.698915 13318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:45:01.699086 13318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:45:01.699280 13318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:01.701668 13318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } }
I20260927 13:45:01.702136 13318 leader_election.cc:302] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125cfb0f8c7f4200a4b02581f451b99e; no voters:
I20260927 13:45:01.702772 13318 leader_election.cc:288] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:45:01.702883 13323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:45:01.703703 13323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [term 1 LEADER]: Becoming Leader. State: Replica: 125cfb0f8c7f4200a4b02581f451b99e, State: Running, Role: LEADER
I20260927 13:45:01.704349 13323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } }
I20260927 13:45:01.704825 13318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:45:01.707505 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125cfb0f8c7f4200a4b02581f451b99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } } }
I20260927 13:45:01.707978 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:01.707639 13325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125cfb0f8c7f4200a4b02581f451b99e. Latest consensus state: current_term: 1 leader_uuid: "125cfb0f8c7f4200a4b02581f451b99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125cfb0f8c7f4200a4b02581f451b99e" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44345 } } }
I20260927 13:45:01.709084 13325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:01.715960 13331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:45:01.721611 13331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:45:01.722239 13318 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:45:01.747840 13331 catalog_manager.cc:1347] Generated new cluster ID: 230a66a086c844eb9996a2f6b25702f5
I20260927 13:45:01.748039 13331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:45:01.783000 13331 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:45:01.785072 13331 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:45:01.803056 13331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: Generated new TSK 0
I20260927 13:45:01.803916 13331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:45:02.466617 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:58323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:02 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:45:02.875535 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:02.875916 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:02.876137 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:02.888010 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:02.888136 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:02.888365 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:45:02.904129 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.237.212:58323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:02.905807 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:02.907562 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:02.923547 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:02.925101 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:02.930135 13363 server_base.cc:1034] running on GCE node
W20260927 13:45:02.930660 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:02.931461 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:02.932608 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:02.933881 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790516702933819 us; error 50 us; skew 500 ppm
I20260927 13:45:02.934257 13363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:02 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:02.938234 13363 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:45:02.940187 13363 webserver.cc:460] Webserver started at http://127.1.237.193:46033/ using document root <none> and password file <none>
I20260927 13:45:02.941125 13363 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:02.941273 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:02.941674 13363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:02 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:02.963044 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "9f757461f09945ef9ce63c6cb4af9c63"
format_stamp: "Formatted at 2026-09-27 13:45:02 on dist-test-slave-7g3l"
server_key: "7e8dfa59cf526b62084d075a901bab51"
server_key_iv: "52f3b903a42ead1c600d889733d8d2f3"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:02.964393 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "9f757461f09945ef9ce63c6cb4af9c63"
format_stamp: "Formatted at 2026-09-27 13:45:02 on dist-test-slave-7g3l"
server_key: "7e8dfa59cf526b62084d075a901bab51"
server_key_iv: "52f3b903a42ead1c600d889733d8d2f3"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:02.971678 13363 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 13:45:03.001291 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:03.003484 13363 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.007s
I20260927 13:45:03.003770 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "9f757461f09945ef9ce63c6cb4af9c63"
format_stamp: "Formatted at 2026-09-27 13:45:02 on dist-test-slave-7g3l"
server_key: "7e8dfa59cf526b62084d075a901bab51"
server_key_iv: "52f3b903a42ead1c600d889733d8d2f3"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:03.004133 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:03.059880 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:03.065099 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:03.065930 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:03.084790 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:03.093443 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:03.093636 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 13:45:03.093848 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:03.093963 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:03.333104 13363 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:34431
I20260927 13:45:03.333122 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:34431 every 8 connection(s)
Sep 27 13:45:03 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:03.335340 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:45:03.345394 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 13363
I20260927 13:45:03.345597 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:45:03.346345 1975 external_mini_cluster.cc:1428] Setting key 54a7d073e578414822672d70ba31817b
Sep 27 13:45:03 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:03.408627 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:03.339287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:44345 (local address 127.1.237.193:32913)
0927 13:45:03.343223 (+ 3936us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:03.343269 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:03.346093 (+ 2824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:03.384691 (+ 38598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:03.384714 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:03.385579 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:03.386495 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:03.386522 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:03.388094 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:03.388103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:03.390409 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:03.390428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:03.391644 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:03.391655 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:03.391998 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:03.392789 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:03.392823 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:03.396219 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:03.403666 (+ 7447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:03.403679 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:03.403701 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:03.404301 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:03.404763 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:03.404771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:03.404778 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:03.405017 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:03.405684 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:03.405696 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:03.406167 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 13:45:03.406461 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 13:45:03.407054 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3622,"thread_start_us":3375,"threads_started":1}
I20260927 13:45:03.410633 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:03.338399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:32913 (local address 127.1.237.254:44345)
0927 13:45:03.383029 (+ 44630us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:03.383035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:03.383060 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:45:03.383128 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:03.383134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:03.383285 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:03.383401 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:03.386713 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:03.387896 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:03.390606 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:03.391367 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:03.392442 (+ 1075us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:03.396495 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:03.396503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:03.396511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:03.396556 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:03.403484 (+ 6928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:03.404464 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:03.404472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:03.404475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:03.404632 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:03.405160 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:03.405167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:03.405172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:03.405523 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:03.409586 (+ 4063us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:03.409734 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 13:45:03.410004 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44480,"thread_start_us":44361,"threads_started":1}
I20260927 13:45:03.412231 13496 heartbeater.cc:347] Connected to a master server at 127.1.237.254:44345
I20260927 13:45:03.412552 13496 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:03.413429 13496 heartbeater.cc:511] Master 127.1.237.254:44345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:45:03.416419 13283 ts_manager.cc:194] Registered new tserver with Master: 9f757461f09945ef9ce63c6cb4af9c63 (127.1.237.193:34431)
I20260927 13:45:03.417804 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:32913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:45:03.447813 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:58323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:03 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516702, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
W20260927 13:45:03.911826 13502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:03.912292 13502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:03.912492 13502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:03.922528 13502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:03.922724 13502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:03.922952 13502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:45:03.938050 13502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.237.212:58323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:03.940033 13502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:03.941767 13502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:03.985208 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:03.986101 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:03.991010 13502 server_base.cc:1034] running on GCE node
W20260927 13:45:03.991492 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:03.992357 13502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:03.993626 13502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:03.994915 13502 hybrid_clock.cc:648] HybridClock initialized: now 1790516703994842 us; error 60 us; skew 500 ppm
I20260927 13:45:03.995337 13502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:03 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516703, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:03.999467 13502 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:45:04.001405 13502 webserver.cc:460] Webserver started at http://127.1.237.194:43139/ using document root <none> and password file <none>
I20260927 13:45:04.002357 13502 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:04.002506 13502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:04.002889 13502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516703, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:04.028993 13502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "dc174ff189dc482c989f2941e105b9e2"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "295720e16ee6b1492f02c65d6da903d2"
server_key_iv: "c04d6499286fdf56d891f21b00c657a8"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.029874 13502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "dc174ff189dc482c989f2941e105b9e2"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "295720e16ee6b1492f02c65d6da903d2"
server_key_iv: "c04d6499286fdf56d891f21b00c657a8"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.035540 13502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:45:04.052268 13526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.054028 13502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:45:04.054253 13502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "dc174ff189dc482c989f2941e105b9e2"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "295720e16ee6b1492f02c65d6da903d2"
server_key_iv: "c04d6499286fdf56d891f21b00c657a8"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.054596 13502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:04.077586 13502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:04.083034 13502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:04.083420 13502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:04.084604 13502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:04.086241 13502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:04.086372 13502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.086550 13502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:04.086660 13502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.132205 13502 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:40795
I20260927 13:45:04.132213 13638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:40795 every 8 connection(s)
I20260927 13:45:04.133522 13502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:45:04.139045 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 13502
I20260927 13:45:04.139245 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:45:04.139778 1975 external_mini_cluster.cc:1428] Setting key 037d0acb44cc9b630528ec77478329f8
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516703, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:04.158659 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.136349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:38779 (local address 127.1.237.254:44345)
0927 13:45:04.137247 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.137251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.138045 (+ 794us) server_negotiation.cc:407] Connection header received
0927 13:45:04.140687 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.140694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.140794 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.140904 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:04.142807 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.143936 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.146496 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.147657 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.147802 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.151525 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:04.151532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:04.151537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:04.151574 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:04.154933 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.155929 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.155934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.155938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.156075 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.156717 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.156723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.156726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.157048 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:04.157197 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.158062 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.158274 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":669,"threads_started":1}
I20260927 13:45:04.160049 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.137093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:44345 (local address 127.1.237.194:38779)
0927 13:45:04.137879 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.137917 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.140112 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.141180 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.141191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.141767 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:04.142605 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.142632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.144097 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.144106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.146323 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.146341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.147886 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.147896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.148232 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.149009 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:04.149043 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:04.151296 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:04.155093 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:04.155105 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:04.155125 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:04.155805 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:04.156198 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:04.156206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:04.156212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:04.156471 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:04.157196 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:04.157207 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:04.157635 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.157928 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.158493 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":365,"threads_started":1}
I20260927 13:45:04.161707 13639 heartbeater.cc:347] Connected to a master server at 127.1.237.254:44345
I20260927 13:45:04.162024 13639 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:04.162842 13639 heartbeater.cc:511] Master 127.1.237.254:44345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
I20260927 13:45:04.164518 13283 ts_manager.cc:194] Registered new tserver with Master: dc174ff189dc482c989f2941e105b9e2 (127.1.237.194:40795)
I20260927 13:45:04.165400 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:38779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:45:04.194180 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:58323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:45:04.421367 13496 heartbeater.cc:503] Master 127.1.237.254:44345 was elected leader, sending a full tablet report...
W20260927 13:45:04.521842 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:04.522112 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:04.522277 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:04.529235 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:04.529330 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:04.529531 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:45:04.539201 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.237.212:58323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:44345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:04.540380 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:04.541580 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:04.554154 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:04.554101 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:04.554227 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:04.555102 13645 server_base.cc:1034] running on GCE node
I20260927 13:45:04.555743 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:04.556697 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:04.557919 13645 hybrid_clock.cc:648] HybridClock initialized: now 1790516704557875 us; error 50 us; skew 500 ppm
I20260927 13:45:04.558226 13645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:04.561762 13645 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:45:04.563216 13645 webserver.cc:460] Webserver started at http://127.1.237.195:43973/ using document root <none> and password file <none>
I20260927 13:45:04.563872 13645 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:04.564007 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:04.564298 13645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:04.578245 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "93ea8d898fcd42e28f52713f6bce91b3"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "d345f29c55101ea3910b6670f7ced60e"
server_key_iv: "65442b069e171f875aabc33855f67373"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.579092 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "93ea8d898fcd42e28f52713f6bce91b3"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "d345f29c55101ea3910b6670f7ced60e"
server_key_iv: "65442b069e171f875aabc33855f67373"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.584298 13645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 13:45:04.598253 13660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.599782 13645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:45:04.599956 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "93ea8d898fcd42e28f52713f6bce91b3"
format_stamp: "Formatted at 2026-09-27 13:45:04 on dist-test-slave-7g3l"
server_key: "d345f29c55101ea3910b6670f7ced60e"
server_key_iv: "65442b069e171f875aabc33855f67373"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:04.600189 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:04.616732 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:04.620468 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:04.620772 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:04.621671 13645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:04.622991 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:04.623070 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.623238 13645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:04.623291 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:04.656875 13645 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:44125
I20260927 13:45:04.657298 13772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:44125 every 8 connection(s)
I20260927 13:45:04.658373 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:45:04.666507 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 13645
I20260927 13:45:04.666718 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:45:04.667325 1975 external_mini_cluster.cc:1428] Setting key f96fd8b67f3a3489bb214c5adde4fc24
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516704, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:04.678894 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.661038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:34375 (local address 127.1.237.254:44345)
0927 13:45:04.661539 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.661545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.662401 (+ 856us) server_negotiation.cc:407] Connection header received
0927 13:45:04.664152 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.664156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.664228 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.664320 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:04.665615 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.666769 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.668927 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.669625 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.669744 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.673638 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:04.673645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:04.673650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:04.673700 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:04.676417 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.677053 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.677056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.677058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.677169 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.677547 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.677550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.677552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.677791 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:04.677890 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.678506 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.678662 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":208,"threads_started":1}
I20260927 13:45:04.679917 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.661592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:44345 (local address 127.1.237.195:34375)
0927 13:45:04.662248 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.662283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.663913 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.664521 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.664528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.664924 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:04.665459 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.665474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.666933 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.666939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.668785 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.668809 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.669848 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.669856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.670142 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.670886 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:04.670909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:04.673447 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:04.676532 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:04.676539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:04.676557 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:04.676944 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:04.677254 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:04.677257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:04.677259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:04.677431 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:04.677894 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:04.677900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:04.678198 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.678408 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.678772 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":286,"threads_started":1}
I20260927 13:45:04.681463 13773 heartbeater.cc:347] Connected to a master server at 127.1.237.254:44345
I20260927 13:45:04.681821 13773 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:04.682583 13773 heartbeater.cc:511] Master 127.1.237.254:44345 requested a full tablet report, sending...
I20260927 13:45:04.684005 13283 ts_manager.cc:194] Registered new tserver with Master: 93ea8d898fcd42e28f52713f6bce91b3 (127.1.237.195:44125)
I20260927 13:45:04.684780 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:34375
I20260927 13:45:04.694983 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:45:04.712167 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.699227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.1.237.254:44345)
0927 13:45:04.699439 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.699448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.699762 (+ 314us) server_negotiation.cc:407] Connection header received
0927 13:45:04.699889 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.699893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.699962 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.700043 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:04.701399 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.702207 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.703626 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.704045 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.704201 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.707063 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:04.707071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:04.707076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:04.707118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:04.709686 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.710386 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.710390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.710392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.710505 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.710824 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.710827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.710829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.711118 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:04.711357 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.711679 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.711838 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:45:04.716033 13283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:45:04.723035 13283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:45:04.759308 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.734951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:40795 (local address 127.0.0.1:37552)
0927 13:45:04.747770 (+ 12819us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.748827 (+ 1057us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.748978 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.750042 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.750047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.750079 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:04.750520 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.750533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.753015 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.753023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.756183 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.756198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.758245 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.758255 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.758480 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.758537 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.758667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.758835 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12691,"mutex_wait_us":135,"thread_start_us":11786,"threads_started":1}
I20260927 13:45:04.759753 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.735588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.1.237.195:44125)
0927 13:45:04.747244 (+ 11656us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.747253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.747289 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:45:04.748007 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.748017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.748318 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.748552 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:04.749780 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.753150 (+ 3370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.757253 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.758644 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.758806 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.759018 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.759268 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.759414 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11451,"thread_start_us":11699,"threads_started":1}
I20260927 13:45:04.759757 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.734863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:44125 (local address 127.0.0.1:49210)
0927 13:45:04.747102 (+ 12239us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.747138 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.747317 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.748951 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.748963 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.749003 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:04.749537 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.749552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.753375 (+ 3823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.753382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.757086 (+ 3704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.757100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.758866 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.758872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.759054 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.759087 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.759225 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.759349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11987,"mutex_wait_us":364,"thread_start_us":12618,"threads_started":1}
I20260927 13:45:04.760392 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.735177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.1.237.194:40795)
0927 13:45:04.747822 (+ 12645us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.747832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.749123 (+ 1291us) server_negotiation.cc:407] Connection header received
0927 13:45:04.749246 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.749257 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.749554 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.749760 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:04.750778 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.752817 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.756340 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.757792 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.757912 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.758109 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.759697 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.760075 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12499,"thread_start_us":12459,"threads_started":1}
I20260927 13:45:04.763473 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.748111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59058)
0927 13:45:04.754222 (+ 6111us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.754248 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.754388 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.755470 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.755475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.755504 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:04.756026 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.756059 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.757562 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.757568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.759618 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.759637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.762604 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.762610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.762784 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.762817 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.762913 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.763023 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5939,"mutex_wait_us":3,"thread_start_us":5757,"threads_started":1}
I20260927 13:45:04.763482 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.748130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.1.237.193:34431)
0927 13:45:04.754892 (+ 6762us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.754900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.754945 (+ 45us) server_negotiation.cc:407] Connection header received
0927 13:45:04.755039 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.755046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.755293 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.755467 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:04.756227 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.757393 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.759799 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.762400 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.762535 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.762769 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.762963 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.763120 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6574,"thread_start_us":6431,"threads_started":1}
I20260927 13:45:04.764006 13574 tablet_service.cc:1467] Processing CreateTablet for tablet 726d8908f516402fa5a0ac1987ad230c (DEFAULT_TABLE table=test-table [id=7ddfad74e15d49aa98f4e8607a3ac373]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:04.764460 13708 tablet_service.cc:1467] Processing CreateTablet for tablet 726d8908f516402fa5a0ac1987ad230c (DEFAULT_TABLE table=test-table [id=7ddfad74e15d49aa98f4e8607a3ac373]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:04.765586 13574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d8908f516402fa5a0ac1987ad230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:04.765920 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d8908f516402fa5a0ac1987ad230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:04.767264 13431 tablet_service.cc:1467] Processing CreateTablet for tablet 726d8908f516402fa5a0ac1987ad230c (DEFAULT_TABLE table=test-table [id=7ddfad74e15d49aa98f4e8607a3ac373]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:04.768601 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726d8908f516402fa5a0ac1987ad230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:04.777442 13792 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Bootstrap starting.
I20260927 13:45:04.780057 13793 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Bootstrap starting.
I20260927 13:45:04.781962 13792 tablet_bootstrap.cc:654] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:04.782433 13794 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Bootstrap starting.
I20260927 13:45:04.783437 13792 log.cc:824] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:04.784328 13793 tablet_bootstrap.cc:654] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:04.785430 13794 tablet_bootstrap.cc:654] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:04.785426 13793 log.cc:824] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:04.786926 13794 log.cc:824] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:04.787250 13792 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: No bootstrap required, opened a new log
I20260927 13:45:04.787520 13792 ts_tablet_manager.cc:1397] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 13:45:04.788852 13793 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: No bootstrap required, opened a new log
I20260927 13:45:04.789135 13793 ts_tablet_manager.cc:1397] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 13:45:04.789945 13794 tablet_bootstrap.cc:492] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: No bootstrap required, opened a new log
I20260927 13:45:04.790203 13794 ts_tablet_manager.cc:1397] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:45:04.794543 13792 raft_consensus.cc:348] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.794904 13792 raft_consensus.cc:374] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:04.795017 13792 raft_consensus.cc:729] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ea8d898fcd42e28f52713f6bce91b3, State: Initialized, Role: FOLLOWER
I20260927 13:45:04.795662 13792 consensus_queue.cc:260] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.796875 13773 heartbeater.cc:503] Master 127.1.237.254:44345 was elected leader, sending a full tablet report...
I20260927 13:45:04.797591 13792 ts_tablet_manager.cc:1428] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 13:45:04.800274 13793 raft_consensus.cc:348] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.801169 13793 raft_consensus.cc:374] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:04.801276 13793 raft_consensus.cc:729] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc174ff189dc482c989f2941e105b9e2, State: Initialized, Role: FOLLOWER
I20260927 13:45:04.801915 13793 consensus_queue.cc:260] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.802073 13794 raft_consensus.cc:348] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.802526 13794 raft_consensus.cc:374] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:04.802608 13794 raft_consensus.cc:729] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f757461f09945ef9ce63c6cb4af9c63, State: Initialized, Role: FOLLOWER
I20260927 13:45:04.803020 13639 heartbeater.cc:503] Master 127.1.237.254:44345 was elected leader, sending a full tablet report...
I20260927 13:45:04.803375 13798 raft_consensus.cc:482] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:45:04.803743 13793 ts_tablet_manager.cc:1428] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 13:45:04.803634 13798 raft_consensus.cc:504] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.804482 13794 consensus_queue.cc:260] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.805404 13798 leader_election.cc:288] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc174ff189dc482c989f2941e105b9e2 (127.1.237.194:40795), 9f757461f09945ef9ce63c6cb4af9c63 (127.1.237.193:34431)
I20260927 13:45:04.806370 13794 ts_tablet_manager.cc:1428] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 13:45:04.813908 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.806898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:55263 (local address 127.1.237.194:40795)
0927 13:45:04.807218 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.807225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.807244 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:45:04.807309 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.807314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.807388 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.807474 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:04.808077 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.809214 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.811588 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.812530 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.812650 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.812701 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.813538 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.813665 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 13:45:04.813884 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.805902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:40795 (local address 127.1.237.195:55263)
0927 13:45:04.806817 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.806839 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.806942 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.807709 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.807716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.807744 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:04.807925 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.807936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.809343 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.809349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.811435 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.811452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.812699 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.812705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.813321 (+ 616us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.813357 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.813469 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.813602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":23}
I20260927 13:45:04.814150 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.807380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:33305 (local address 127.1.237.193:34431)
0927 13:45:04.807553 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.807558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.807577 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:45:04.807770 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.807778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.807840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.807930 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:04.808616 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.809634 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.811912 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.812837 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.813382 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.813468 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.813783 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.813919 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:45:04.814595 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.806403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.1.237.195:33305)
0927 13:45:04.807485 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:04.807507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:04.807634 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:04.808087 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:04.808091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:04.808116 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:04.808445 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.808456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.809781 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.809785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.811767 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:04.811780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.813427 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.813431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:04.813618 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.813651 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:45:04.813743 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 13:45:04.813862 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":21,"thread_start_us":2934,"threads_started":1}
I20260927 13:45:04.814888 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d8908f516402fa5a0ac1987ad230c" candidate_uuid: "93ea8d898fcd42e28f52713f6bce91b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc174ff189dc482c989f2941e105b9e2" is_pre_election: true
I20260927 13:45:04.815398 13594 raft_consensus.cc:2454] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93ea8d898fcd42e28f52713f6bce91b3 in term 0.
I20260927 13:45:04.815562 13451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d8908f516402fa5a0ac1987ad230c" candidate_uuid: "93ea8d898fcd42e28f52713f6bce91b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f757461f09945ef9ce63c6cb4af9c63" is_pre_election: true
I20260927 13:45:04.816084 13451 raft_consensus.cc:2454] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93ea8d898fcd42e28f52713f6bce91b3 in term 0.
I20260927 13:45:04.816042 13664 leader_election.cc:302] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93ea8d898fcd42e28f52713f6bce91b3, dc174ff189dc482c989f2941e105b9e2; no voters:
I20260927 13:45:04.816687 13798 raft_consensus.cc:2781] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:45:04.816824 13798 raft_consensus.cc:482] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:45:04.816920 13798 raft_consensus.cc:3037] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:04.819099 13798 raft_consensus.cc:504] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.819792 13798 leader_election.cc:288] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [CANDIDATE]: Term 1 election: Requested vote from peers dc174ff189dc482c989f2941e105b9e2 (127.1.237.194:40795), 9f757461f09945ef9ce63c6cb4af9c63 (127.1.237.193:34431)
I20260927 13:45:04.820200 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d8908f516402fa5a0ac1987ad230c" candidate_uuid: "93ea8d898fcd42e28f52713f6bce91b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc174ff189dc482c989f2941e105b9e2"
I20260927 13:45:04.820200 13451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726d8908f516402fa5a0ac1987ad230c" candidate_uuid: "93ea8d898fcd42e28f52713f6bce91b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f757461f09945ef9ce63c6cb4af9c63"
I20260927 13:45:04.820390 13451 raft_consensus.cc:3037] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:04.820387 13594 raft_consensus.cc:3037] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:04.823484 13451 raft_consensus.cc:2454] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93ea8d898fcd42e28f52713f6bce91b3 in term 1.
I20260927 13:45:04.823484 13594 raft_consensus.cc:2454] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93ea8d898fcd42e28f52713f6bce91b3 in term 1.
I20260927 13:45:04.823834 13664 leader_election.cc:302] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93ea8d898fcd42e28f52713f6bce91b3, dc174ff189dc482c989f2941e105b9e2; no voters:
I20260927 13:45:04.824107 13798 raft_consensus.cc:2781] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:45:04.824538 13798 raft_consensus.cc:686] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 1 LEADER]: Becoming Leader. State: Replica: 93ea8d898fcd42e28f52713f6bce91b3, State: Running, Role: LEADER
I20260927 13:45:04.825023 13798 consensus_queue.cc:237] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } }
I20260927 13:45:04.829286 13283 catalog_manager.cc:5495] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93ea8d898fcd42e28f52713f6bce91b3 (127.1.237.195). New cstate: current_term: 1 leader_uuid: "93ea8d898fcd42e28f52713f6bce91b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93ea8d898fcd42e28f52713f6bce91b3" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 44125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 } health_report { overall_health: UNKNOWN } } }
I20260927 13:45:04.850243 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.844973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.1.237.195:44125)
0927 13:45:04.845161 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.845165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.845181 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:45:04.845330 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.845334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.845413 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.845533 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:45:04.845867 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.846598 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.848060 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.848565 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.848671 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.849158 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:45:04.849649 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:45:04.849757 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.849888 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.849947 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 13:45:04.857998 13451 raft_consensus.cc:1261] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 1 FOLLOWER]: Refusing update from remote peer 93ea8d898fcd42e28f52713f6bce91b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:45:04.858022 13594 raft_consensus.cc:1261] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 1 FOLLOWER]: Refusing update from remote peer 93ea8d898fcd42e28f52713f6bce91b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:45:04.858908 13802 consensus_queue.cc:1035] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f757461f09945ef9ce63c6cb4af9c63" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 13:45:04.859766 13798 consensus_queue.cc:1035] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc174ff189dc482c989f2941e105b9e2" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:45:04.887763 13283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:45:04.888033 13283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:45:04.891665 13283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 125cfb0f8c7f4200a4b02581f451b99e: Sending DeleteTablet for 3 replicas of tablet 726d8908f516402fa5a0ac1987ad230c
I20260927 13:45:04.892800 13574 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d8908f516402fa5a0ac1987ad230c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:04 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:37552
I20260927 13:45:04.893033 13431 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d8908f516402fa5a0ac1987ad230c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:04 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:59058
I20260927 13:45:04.893107 13708 tablet_service.cc:1514] Processing DeleteTablet for tablet 726d8908f516402fa5a0ac1987ad230c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:04 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:49210
I20260927 13:45:04.894649 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 13363
I20260927 13:45:04.903141 13817 tablet_replica.cc:330] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: stopping tablet replica
I20260927 13:45:04.903554 13817 raft_consensus.cc:2229] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:45:04.904448 13817 raft_consensus.cc:2258] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:45:04.906951 13817 ts_tablet_manager.cc:1905] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:45:04.907647 13815 tablet_replica.cc:330] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: stopping tablet replica
I20260927 13:45:04.908020 13815 raft_consensus.cc:2229] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:45:04.908474 13815 raft_consensus.cc:2258] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:45:04.909135 13816 tablet_replica.cc:330] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: stopping tablet replica
I20260927 13:45:04.909479 13816 raft_consensus.cc:2229] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:45:04.909957 13816 raft_consensus.cc:2258] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:45:04.913071 13815 ts_tablet_manager.cc:1905] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:45:04 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:45:04.919258 13817 ts_tablet_manager.cc:1918] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:45:04.919416 13817 log.cc:1196] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/726d8908f516402fa5a0ac1987ad230c
I20260927 13:45:04.919606 13815 ts_tablet_manager.cc:1918] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:45:04.919752 13815 log.cc:1196] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/726d8908f516402fa5a0ac1987ad230c
I20260927 13:45:04.920012 13817 ts_tablet_manager.cc:1939] T 726d8908f516402fa5a0ac1987ad230c P 93ea8d898fcd42e28f52713f6bce91b3: Deleting consensus metadata
I20260927 13:45:04.920224 13815 ts_tablet_manager.cc:1939] T 726d8908f516402fa5a0ac1987ad230c P dc174ff189dc482c989f2941e105b9e2: Deleting consensus metadata
I20260927 13:45:04.921857 13267 catalog_manager.cc:4841] TS 93ea8d898fcd42e28f52713f6bce91b3 (127.1.237.195:44125): tablet 726d8908f516402fa5a0ac1987ad230c (table test-table [id=7ddfad74e15d49aa98f4e8607a3ac373]) successfully deleted
I20260927 13:45:04.921936 13270 catalog_manager.cc:4841] TS dc174ff189dc482c989f2941e105b9e2 (127.1.237.194:40795): tablet 726d8908f516402fa5a0ac1987ad230c (table test-table [id=7ddfad74e15d49aa98f4e8607a3ac373]) successfully deleted
I20260927 13:45:04.922356 13816 ts_tablet_manager.cc:1905] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:45:04.927616 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:04.896382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.1.237.193:34431)
0927 13:45:04.896606 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:04.896615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:04.907178 (+ 10563us) server_negotiation.cc:407] Connection header received
0927 13:45:04.907738 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:04.907747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:04.907853 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:04.907942 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:04.910175 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.911376 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.913452 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:04.914090 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:04.914210 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:04.919840 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:04.919856 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:04.919876 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:04.919920 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:04.923869 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.925093 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.925101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.925104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.925278 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:04.925716 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:04.925722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:04.925728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:04.926211 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:04.926546 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:04.926882 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 13:45:04.927194 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:45:04.928289 13491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:05.017055 13816 ts_tablet_manager.cc:1918] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:45:05.017326 13816 log.cc:1196] T 726d8908f516402fa5a0ac1987ad230c P 9f757461f09945ef9ce63c6cb4af9c63: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/726d8908f516402fa5a0ac1987ad230c
I20260927 13:45:05.017750 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 13363
W20260927 13:45:05.034770 13267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:34431 (error 108)
I20260927 13:45:05.034941 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 13502
I20260927 13:45:05.037189 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.036693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59072)
0927 13:45:05.036877 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.036991 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 13:45:05.037482 13267 catalog_manager.cc:4584] TS 9f757461f09945ef9ce63c6cb4af9c63 (127.1.237.193:34431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 726d8908f516402fa5a0ac1987ad230c: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Sep 27 13:45:05 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:45:05.051975 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.035724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.1.237.194:40795)
0927 13:45:05.035914 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:05.035922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:05.035994 (+ 72us) server_negotiation.cc:407] Connection header received
0927 13:45:05.036235 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:05.036241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:05.036330 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:05.036435 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:05.038223 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:05.039351 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:05.040570 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:05.041177 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:05.041322 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:05.045376 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:05.045390 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:05.045409 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:05.045450 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:05.049224 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:05.049842 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:05.049851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:05.049855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:05.050036 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:05.050392 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:05.050397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:05.050401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:05.050858 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:05.051186 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:05.051382 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 13:45:05.051601 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:45:05.052667 13634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:05.091177 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.090614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59088)
0927 13:45:05.090843 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.090954 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 13:45:05.092242 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.091866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59096)
0927 13:45:05.092014 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.092104 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 13:45:05.139830 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 13502
I20260927 13:45:05.153054 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.152479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59106)
0927 13:45:05.152729 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.152840 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:45:05.154225 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.153805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59120)
0927 13:45:05.153976 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.154080 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:45:05.157167 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 13645
Sep 27 13:45:05 dist-test-slave-7g3l krb5kdc[12807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:45:05.173432 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.157970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.1.237.195:44125)
0927 13:45:05.158193 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:05.158199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:05.158228 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:45:05.158354 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:05.158359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:05.158429 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:05.158550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:05.159875 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:05.160717 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:05.162010 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:05.162439 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:05.162597 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:05.167229 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:05.167244 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:05.167259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:05.167301 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:05.170358 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:05.171391 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:05.171403 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:05.171410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:05.171589 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:05.172038 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:05.172041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:05.172046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:05.172383 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:05.172614 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:05.172975 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 13:45:05.173128 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:45:05.174151 13768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:05.234776 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.234338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59134)
0927 13:45:05.234534 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.234624 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 13:45:05.235881 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.235521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59146)
0927 13:45:05.235704 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.235783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 13:45:05.294493 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 13645
I20260927 13:45:05.313195 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 13251
I20260927 13:45:05.314323 13313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:05.424155 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:05.423398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34431 (local address 127.0.0.1:59148)
0927 13:45:05.423810 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:05.423903 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260927 13:45:05.424345 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 13251
I20260927 13:45:05.447028 1975 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:45:05.781441 1975 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:45:05.781665 1975 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:45:06.126807 1975 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:45:06.127 UTC [12837] LOG: received smart shutdown request
2026-09-27 13:45:06.130 UTC [12837] LOG: background worker "logical replication launcher" (PID 12846) exited with exit code 1
2026-09-27 13:45:06.130 UTC [12841] LOG: shutting down
2026-09-27 13:45:06.278 UTC [12837] LOG: database system is shut down
2026-09-27T13:45:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45034 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:45:06 dist-test-slave-7g3l krb5kdc[13832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:45:06 dist-test-slave-7g3l krb5kdc[13832](info): set up 2 sockets
Sep 27 13:45:06 dist-test-slave-7g3l krb5kdc[13832](info): commencing operation
krb5kdc: starting...
W20260927 13:45:07.475596 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:45:07 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:45:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangeradmin_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangeradmin_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangerlookup_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangerlookup_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
I20260927 13:45:07.770026 1975 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskijlVCN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/postgres -l logfile start
2026-09-27 13:45:10.033 UTC [13862] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 13:45:10.033 UTC [13862] LOG: listening on IPv4 address "127.1.237.212", port 39795
2026-09-27 13:45:10.036 UTC [13862] LOG: listening on Unix socket "/tmp/.s.PGSQL.39795"
2026-09-27 13:45:10.057 UTC [13865] LOG: database system was shut down at 2026-09-27 13:45:08 UTC
2026-09-27 13:45:10.060 UTC [13862] LOG: database system is ready to accept connections
I20260927 13:45:11.027287 1975 mini_postgres.cc:96] Postgres bound to 39795
2026-09-27 13:45:11.044 UTC [13873] FATAL: database "slave" does not exist
127.1.237.212:39795 - accepting connections
I20260927 13:45:11.044900 1975 mini_ranger.cc:161] Starting Ranger...
I20260927 13:45:11.070468 1975 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 13:45:11.484149 1975 mini_ranger.cc:87] Created ranger Postgres database
I20260927 13:45:11.484261 1975 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin
2026-09-27 13:45:11,757 [I] DB FLAVOR :POSTGRES
2026-09-27 13:45:11,758 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:45:11,758 [I] Checking connection..
2026-09-27 13:45:11,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 13:45:12,002 [I] Checking connection passed.
2026-09-27 13:45:12,002 [I] --------- Verifying version history table ---------
2026-09-27 13:45:12,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:45:12,249 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:45:12,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:45:12,505 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 13:45:12,506 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 13:45:12,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 13:45:12.763 UTC [13937] WARNING: there is no transaction in progress
2026-09-27 13:45:12,769 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 13:45:12,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 13:45:13,024 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 13:45:13,024 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:45:13,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 13:45:13,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 13:45:13,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7g3l.c.gcp-upstream.internal','N') ;"
2026-09-27 13:45:13,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:45:13,966 [I] Table x_portal_user does not exist in database ranger
2026-09-27 13:45:13,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:45:14,193 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 13:45:14,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:45:14,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:45:14,675 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 13:45:14,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 13:45:15.564 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.579 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.593 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.607 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.621 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.637 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.687 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.693 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.707 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.716 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.725 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.737 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.746 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.751 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.757 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:15.763 UTC [14045] WARNING: there is no transaction in progress
2026-09-27 13:45:16,321 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 13:45:16,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 13:45:16,566 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 13:45:16,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 13:45:16,807 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 13:45:16,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 13:45:17,055 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 13:45:17,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 13:45:17,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 13:45:17,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7g3l.c.gcp-upstream.internal';"
2026-09-27 13:45:17,760 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 13:45:17,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 13:45:18,005 [I] DB_PATCHES have already been applied
I20260927 13:45:18.011456 1975 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskijlVCN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 13:45:18.011533 1975 mini_ranger.cc:193] Using host: 127.1.237.212
I20260927 13:45:18.025159 1975 mini_ranger.cc:227] Ranger admin URL: http://127.1.237.212:47763
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 1:45:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 13:45:19.126210 1975 mini_ranger.cc:160] Time spent starting Ranger: real 8.081s user 0.004s sys 0.033s
Sep 27, 2026 1:45:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 1:45:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 1:45:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 1:45:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 1:45:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 13:45:40.544981 1975 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.237.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.237.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangerkms_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.237.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/rangerkms_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.237.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.237.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.237.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/HTTP_127.1.237.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 13:45:40 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516740, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 13:45:40.722137 1975 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 13:45:40.743427 1975 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 13:45:41.516009 1975 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 13:45:41.516121 1975 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms
I20260927 13:45:41.516178 1975 mini_ranger_kms.cc:227] Using postgres at 127.1.237.212:39795
2026-09-27 13:45:41,869 [I] DB FLAVOR :POSTGRES
2026-09-27 13:45:41,869 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 13:45:41,869 [I] Checking connection
2026-09-27 13:45:41,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 13:45:42,103 [I] connection success
2026-09-27 13:45:42,103 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 13:45:42,103 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 13:45:42,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 13:45:42,324 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 13:45:42,324 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 13:45:42,324 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 13:45:42,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.237.212:39795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 13:45:42,566 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 13:45:43.041169 1975 mini_ranger_kms.cc:310] Created kms service
I20260927 13:45:43.220939 1975 mini_ranger_kms.cc:326] Created kudu user
I20260927 13:45:43.263638 1975 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 13:45:43 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516740, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:43.632359 1975 mini_ranger_kms.cc:384] Added ranger policy
I20260927 13:45:43.632678 1975 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms:/tmp/dist-test-taskijlVCN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskijlVCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 13:45:43.632731 1975 mini_ranger_kms.cc:240] Using host: 127.1.237.212
I20260927 13:45:43.649636 1975 mini_ranger_kms.cc:276] Ranger KMS PID: 14225
I20260927 13:45:43.649765 1975 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.237.212:50523
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 1:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 13:45:45 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790516745, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:45.937666 1975 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.216s user 0.064s sys 0.307s
I20260927 13:45:45.937924 1975 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 13:45:45.938090 1975 mini_ranger_kms.cc:401] 127.1.237.212:50523/kms/v1/keys
Sep 27 13:45:46 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516745, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
Sep 27 13:45:46 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516745, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.237.212@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
Sep 27 13:45:48 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:45:48.582297 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38635
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:38635
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:45:48.899947 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:48.900200 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:48.900337 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:48.907335 14276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:48.907433 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:48.907500 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:48.907600 14276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:48.907666 14276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:48.918058 14276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.237.212:50523/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:38635
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:48.919569 14276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:48.920923 14276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:48.934335 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:48.934409 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:48.934966 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:48.935076 14276 server_base.cc:1034] running on GCE node
I20260927 13:45:48.936077 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:48.937826 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:48.939073 14276 hybrid_clock.cc:648] HybridClock initialized: now 1790516748939029 us; error 33 us; skew 500 ppm
I20260927 13:45:48.939458 14276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:48 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:48.950523 14276 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:45:48.952221 14276 webserver.cc:460] Webserver started at http://127.1.237.254:32827/ using document root <none> and password file <none>
I20260927 13:45:48.952940 14276 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:48.953063 14276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:48.953403 14276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:48 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:49.010897 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "5c605b19363a4a97aa79e265ec4ed192"
format_stamp: "Formatted at 2026-09-27 13:45:49 on dist-test-slave-7g3l"
server_key: "8be24317b5b76e2089fe6e5d16446856"
server_key_iv: "df5709cda8f646638cdb5856615f7bff"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:49.011905 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "5c605b19363a4a97aa79e265ec4ed192"
format_stamp: "Formatted at 2026-09-27 13:45:49 on dist-test-slave-7g3l"
server_key: "8be24317b5b76e2089fe6e5d16446856"
server_key_iv: "df5709cda8f646638cdb5856615f7bff"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:49.017330 14276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:45:49.032697 14291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:49.034158 14276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:45:49.034349 14276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "5c605b19363a4a97aa79e265ec4ed192"
format_stamp: "Formatted at 2026-09-27 13:45:49 on dist-test-slave-7g3l"
server_key: "8be24317b5b76e2089fe6e5d16446856"
server_key_iv: "df5709cda8f646638cdb5856615f7bff"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:49.034559 14276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:49.052001 14276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:49.055804 14276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:49.056100 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:49.074977 14276 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:38635
I20260927 13:45:49.074981 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:38635 every 8 connection(s)
I20260927 13:45:49.076303 14276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:45:49.080700 14343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:49.081707 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14276
I20260927 13:45:49.081938 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:45:49.082576 1975 external_mini_cluster.cc:1428] Setting key a1c8693d9f9d440aa3d444773c6e427c
I20260927 13:45:49.094650 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Bootstrap starting.
I20260927 13:45:49.098385 14343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:49.099639 14343 log.cc:824] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:49.103260 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: No bootstrap required, opened a new log
Sep 27 13:45:49 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:49.113909 14343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:49.114369 14343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:49.114490 14343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c605b19363a4a97aa79e265ec4ed192, State: Initialized, Role: FOLLOWER
I20260927 13:45:49.115284 14343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:49.115542 14343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:45:49.115674 14343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:45:49.115844 14343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:49.118852 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:49.084736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.1.237.254:38635)
0927 13:45:49.100267 (+ 15531us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:49.100281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:49.100332 (+ 51us) server_negotiation.cc:407] Connection header received
0927 13:45:49.102248 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:49.102269 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:49.102619 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:49.102996 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:49.104393 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:49.105622 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:49.106892 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:49.107588 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:49.107787 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:49.111446 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:49.111456 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:49.111471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:49.111513 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:49.114617 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:49.115400 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:49.115411 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:49.115421 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:49.115645 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:49.116188 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:49.116193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:49.116197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:49.116736 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:49.117060 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:49.117508 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 13:45:49.117915 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15305,"thread_start_us":15075,"threads_started":1}
I20260927 13:45:49.118753 14343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:49.119484 14343 leader_election.cc:302] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c605b19363a4a97aa79e265ec4ed192; no voters:
I20260927 13:45:49.120203 14343 leader_election.cc:288] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:45:49.120398 14348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:45:49.122880 14348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 LEADER]: Becoming Leader. State: Replica: 5c605b19363a4a97aa79e265ec4ed192, State: Running, Role: LEADER
I20260927 13:45:49.123492 14348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:49.124015 14343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:45:49.126423 14350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c605b19363a4a97aa79e265ec4ed192. Latest consensus state: current_term: 1 leader_uuid: "5c605b19363a4a97aa79e265ec4ed192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } } }
I20260927 13:45:49.126798 14350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:49.126525 14349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c605b19363a4a97aa79e265ec4ed192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } } }
I20260927 13:45:49.126925 14349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:49.140753 14353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:45:49.146488 14353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:45:49.165887 14353 catalog_manager.cc:1347] Generated new cluster ID: 47904ee51e2b402baf0f037a05dc1c46
I20260927 13:45:49.166009 14353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:45:49.181298 14343 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 13:45:49.185796 14353 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:45:49.188282 14353 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:45:49.206296 14353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Generated new TSK 0
I20260927 13:45:49.207604 14353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:45:49.984186 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:50 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:45:50.365216 14388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:50.365585 14388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:50.365789 14388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:50.377169 14388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:50.377312 14388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:50.377556 14388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:45:50.388230 14388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:50.389608 14388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:50.391052 14388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:50.406044 14398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:50.407580 14399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:50.415776 14401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:50.415984 14388 server_base.cc:1034] running on GCE node
I20260927 13:45:50.416806 14388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:50.418992 14388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:50.420308 14388 hybrid_clock.cc:648] HybridClock initialized: now 1790516750420227 us; error 66 us; skew 500 ppm
I20260927 13:45:50.420694 14388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:50 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:50.424741 14388 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:45:50.433310 14388 webserver.cc:460] Webserver started at http://127.1.237.193:37693/ using document root <none> and password file <none>
I20260927 13:45:50.434123 14388 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:50.434270 14388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:50.434599 14388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:50 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:50.459899 14388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "6875036f044a475bb1c247f6b3c8d026"
format_stamp: "Formatted at 2026-09-27 13:45:50 on dist-test-slave-7g3l"
server_key: "82f5d32396c5868e6812929f0c82ee3e"
server_key_iv: "94414f0b1a654e8fcfee48f27628ae74"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:50.467916 14388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "6875036f044a475bb1c247f6b3c8d026"
format_stamp: "Formatted at 2026-09-27 13:45:50 on dist-test-slave-7g3l"
server_key: "82f5d32396c5868e6812929f0c82ee3e"
server_key_iv: "94414f0b1a654e8fcfee48f27628ae74"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:50.478480 14388 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260927 13:45:50.507776 14408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:50.509721 14388 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.010s
I20260927 13:45:50.510031 14388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "6875036f044a475bb1c247f6b3c8d026"
format_stamp: "Formatted at 2026-09-27 13:45:50 on dist-test-slave-7g3l"
server_key: "82f5d32396c5868e6812929f0c82ee3e"
server_key_iv: "94414f0b1a654e8fcfee48f27628ae74"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:50.510387 14388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:50.541571 14388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:50.546362 14388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:50.546690 14388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:50.547817 14388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:50.549459 14388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:50.549553 14388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:50.549669 14388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:50.549729 14388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:50.605859 14388 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:38731
I20260927 13:45:50.605898 14520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:38731 every 8 connection(s)
I20260927 13:45:50.607578 14388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:45:50.614102 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14388
I20260927 13:45:50.614353 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:45:50.615398 1975 external_mini_cluster.cc:1428] Setting key a8dff909bcefaca44238b8b526a8c414
Sep 27 13:45:50 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
I20260927 13:45:50.667807 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:50.610447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:39107 (local address 127.1.237.254:38635)
0927 13:45:50.629300 (+ 18853us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:50.629311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:50.645319 (+ 16008us) server_negotiation.cc:407] Connection header received
0927 13:45:50.649173 (+ 3854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:50.649180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:50.649273 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:50.649360 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:50.651794 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:50.652938 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:50.655435 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:50.656062 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:50.656176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:50.660592 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:50.660601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:50.660608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:50.660645 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:50.664239 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:50.665104 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:50.665109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:50.665113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:50.665249 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:50.665758 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:50.665764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:50.665767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:50.666097 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:50.666216 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:50.667191 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 13:45:50.667423 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18675,"thread_start_us":18865,"threads_started":1}
I20260927 13:45:50.670181 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:50.611337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.193:39107)
0927 13:45:50.644665 (+ 33328us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:50.644720 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:50.648794 (+ 4074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:50.649979 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:50.649993 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:50.650646 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:50.651583 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:50.651610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:50.653162 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:50.653168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:50.655270 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:50.655285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:50.656509 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:50.656518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:50.656811 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:50.657608 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:50.657639 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:50.660332 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:50.664395 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:50.664405 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:50.664423 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:50.664961 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:50.665402 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:50.665407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:50.665411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:50.665634 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:50.666303 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:50.666311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:50.666722 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 13:45:50.668310 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0927 13:45:50.668825 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32487,"thread_start_us":32216,"threads_started":1}
I20260927 13:45:50.672252 14521 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:50.672562 14521 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:50.673316 14521 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:50.675828 14308 ts_manager.cc:194] Registered new tserver with Master: 6875036f044a475bb1c247f6b3c8d026 (127.1.237.193:38731)
I20260927 13:45:50.677752 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:39107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:45:50.705147 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:50 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516750, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
W20260927 13:45:51.216035 14527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:51.216504 14527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:51.216697 14527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:51.227546 14527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:51.227735 14527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:51.227989 14527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:45:51.242792 14527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:51.244565 14527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:51.246261 14527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:51.278965 14534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:51.281432 14535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:51.287397 14538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:51.287863 14527 server_base.cc:1034] running on GCE node
I20260927 13:45:51.288722 14527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:51.303457 14527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:51.304785 14527 hybrid_clock.cc:648] HybridClock initialized: now 1790516751304683 us; error 98 us; skew 500 ppm
I20260927 13:45:51.305225 14527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:51.309378 14527 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:45:51.343572 14527 webserver.cc:460] Webserver started at http://127.1.237.194:43921/ using document root <none> and password file <none>
I20260927 13:45:51.344630 14527 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:51.344866 14527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:51.345290 14527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:51.365516 14527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "21bc0d94362c422a93928f5919382762"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "653a6fcc5dc84729839876e923dfb97a"
server_key_iv: "ee3fdc4fc8680e8feb0fc77a76367f75"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.366511 14527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "21bc0d94362c422a93928f5919382762"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "653a6fcc5dc84729839876e923dfb97a"
server_key_iv: "ee3fdc4fc8680e8feb0fc77a76367f75"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.372251 14527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 13:45:51.386019 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.387521 14527 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 13:45:51.387686 14527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "21bc0d94362c422a93928f5919382762"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "653a6fcc5dc84729839876e923dfb97a"
server_key_iv: "ee3fdc4fc8680e8feb0fc77a76367f75"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.387933 14527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:51.405272 14527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:51.408973 14527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:51.409245 14527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:51.410092 14527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:51.411405 14527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:51.411502 14527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.411638 14527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:51.411710 14527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.460227 14527 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:36693
I20260927 13:45:51.460397 14663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:36693 every 8 connection(s)
I20260927 13:45:51.461633 14527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:45:51.469211 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14527
I20260927 13:45:51.469406 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:45:51.469817 1975 external_mini_cluster.cc:1428] Setting key 4f1045e677e26d03a9b25cc309f59350
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:51.484107 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:51.464353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:58969 (local address 127.1.237.254:38635)
0927 13:45:51.464834 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:51.464842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:51.465781 (+ 939us) server_negotiation.cc:407] Connection header received
0927 13:45:51.468219 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:51.468223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:51.468302 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:51.468395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:51.470195 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.471063 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.473384 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.473755 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.473863 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:51.477968 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:51.477972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:51.477975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:51.478008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:51.481081 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:51.481681 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:51.481689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:51.481694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:51.481837 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:51.482205 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:51.482211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:51.482217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:51.482607 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:51.482748 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:51.483391 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 13:45:51.483625 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260927 13:45:51.484647 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:51.464886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.194:58969)
0927 13:45:51.465604 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:51.465653 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:51.467950 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:51.468645 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:51.468657 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:51.469258 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:51.470017 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:51.470043 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.471247 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.471251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:51.473227 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:51.473245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.473984 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.473993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:51.474327 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:51.475029 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:51.475061 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:51.477749 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:51.481206 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:51.481211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:51.481223 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:51.481565 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:51.481929 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:51.481932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:51.481934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:51.482086 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:51.482720 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:51.482726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:51.483001 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 13:45:51.483261 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 13:45:51.483602 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260927 13:45:51.486063 14664 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:51.486344 14664 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:51.487124 14664 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:51.488559 14308 ts_manager.cc:194] Registered new tserver with Master: 21bc0d94362c422a93928f5919382762 (127.1.237.194:36693)
I20260927 13:45:51.489317 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:58969
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:45:51.520525 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:45:51.680648 14521 heartbeater.cc:503] Master 127.1.237.254:38635 was elected leader, sending a full tablet report...
W20260927 13:45:51.828132 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:51.828374 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:51.828459 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:51.835336 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:51.835422 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:51.835520 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:45:51.844842 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:51.846069 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:51.847262 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:51.859213 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:51.859287 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:51.859251 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:51.859985 14670 server_base.cc:1034] running on GCE node
I20260927 13:45:51.860517 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:51.861339 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:51.862578 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790516751862534 us; error 57 us; skew 500 ppm
I20260927 13:45:51.862880 14670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:51.865818 14670 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:45:51.867120 14670 webserver.cc:460] Webserver started at http://127.1.237.195:42319/ using document root <none> and password file <none>
I20260927 13:45:51.867858 14670 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:51.867976 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:51.868258 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:51.883720 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "c37616d4061b4d3fa8323d9cfb849ef9"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "23ca0a83a3357c2f24b9cf4ece13238d"
server_key_iv: "02504de376f2ae523388584a0f9620cb"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.884505 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "c37616d4061b4d3fa8323d9cfb849ef9"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "23ca0a83a3357c2f24b9cf4ece13238d"
server_key_iv: "02504de376f2ae523388584a0f9620cb"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.889570 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:45:51.906138 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.907847 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:45:51.908025 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c37616d4061b4d3fa8323d9cfb849ef9"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "23ca0a83a3357c2f24b9cf4ece13238d"
server_key_iv: "02504de376f2ae523388584a0f9620cb"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:51.908294 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:51.936467 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:51.940361 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:51.940596 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:51.941437 14670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:51.942626 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:51.942694 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.942773 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:51.942819 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:51.974426 14670 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34429
I20260927 13:45:51.974511 14797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34429 every 8 connection(s)
I20260927 13:45:51.975672 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:45:51.979538 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14670
I20260927 13:45:51.979718 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:45:51.980279 1975 external_mini_cluster.cc:1428] Setting key 09e020a9891f56050e93e564e43909a7
Sep 27 13:45:51 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516751, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:51.993952 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:51.978131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:55221 (local address 127.1.237.254:38635)
0927 13:45:51.978306 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:51.978310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:51.979575 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 13:45:51.981657 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:51.981663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:51.981766 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:51.981872 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:51.983269 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.983970 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.985369 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.985748 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.985845 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:51.988811 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:51.988815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:51.988818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:51.988854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:51.991321 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:51.991912 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:51.991915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:51.991917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:51.992032 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:51.992463 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:51.992467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:51.992469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:51.992723 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:51.992839 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:51.993406 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 13:45:51.993634 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:45:51.994735 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:51.978624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.195:55221)
0927 13:45:51.979381 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:51.979438 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:51.981408 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:51.982086 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:51.982093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:51.982464 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:51.983078 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:51.983109 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.984092 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.984096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:51.985249 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:51.985260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:51.985948 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:51.985953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:51.986140 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:51.986715 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:51.986756 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:51.988635 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:51.991433 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:51.991438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:51.991449 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:51.991809 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:51.992144 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:51.992152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:51.992156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:51.992332 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:51.992820 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:51.992825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:51.993074 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 13:45:51.993275 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 13:45:51.993600 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260927 13:45:51.996027 14798 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:51.996309 14798 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:51.996961 14798 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:51.998184 14308 ts_manager.cc:194] Registered new tserver with Master: c37616d4061b4d3fa8323d9cfb849ef9 (127.1.237.195:34429)
I20260927 13:45:51.998962 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:55221
I20260927 13:45:52.007400 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:45:52.008992 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14388
Sep 27 13:45:52 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:45:52.024617 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:52.009886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.1.237.193:38731)
0927 13:45:52.010471 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:52.010477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:52.010499 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:45:52.010551 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:52.010557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:52.010722 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:52.010833 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:52.012257 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.013197 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.014584 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.015230 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.015321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:52.019375 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:52.019385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:52.019397 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:52.019427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:52.021889 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.022538 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.022548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.022557 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.022695 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.023062 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.023069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.023072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.023436 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:52.023734 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:52.024041 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:45:52.024292 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":299,"threads_started":1}
I20260927 13:45:52.025861 14516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:52.105017 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14388
I20260927 13:45:52.120018 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14527
Sep 27 13:45:52 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:45:52.135485 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:52.120916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.1.237.194:36693)
0927 13:45:52.121432 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:52.121438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:52.121463 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:45:52.121525 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:52.121531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:52.121747 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:52.121877 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:52.122952 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.124016 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.125391 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.125911 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.126005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:52.130028 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:52.130038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:52.130050 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:52.130078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:52.132897 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.133706 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.133713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.133715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.133842 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.134138 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.134140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.134142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.134440 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:52.134644 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:52.134907 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 13:45:52.135076 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260927 13:45:52.136634 14659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:52.214986 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14527
I20260927 13:45:52.230418 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14670
Sep 27 13:45:52 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:45:52.245204 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:52.231335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.1.237.195:34429)
0927 13:45:52.231891 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:52.231898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:52.231926 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:45:52.231981 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:52.231986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:52.232203 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:52.232381 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:52.233571 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.234538 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.235852 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:52.236367 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:52.236456 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:52.240525 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:52.240534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:52.240547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:52.240575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:52.242950 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.243629 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.243635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.243637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.243751 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:52.244043 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:52.244046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:52.244048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:52.244349 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:52.244566 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:52.244768 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 13:45:52.244930 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":277,"threads_started":1}
I20260927 13:45:52.246385 14793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:52.325522 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14670
I20260927 13:45:52.341068 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14276
I20260927 13:45:52.342141 14338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:52.442190 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14276
I20260927 13:45:52.461930 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38635
--webserver_interface=127.1.237.254
--webserver_port=32827
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:38635
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:45:52.766969 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:52.767285 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:52.767447 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:52.773904 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:52.774022 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:52.774117 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:52.774230 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:52.774298 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:52.786267 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:38635
--ranger_config_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:52.787837 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:52.789021 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:52.800254 14817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:52.800369 14815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:52.800366 14814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:52.801115 14809 server_base.cc:1034] running on GCE node
I20260927 13:45:52.801868 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:52.803197 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:52.804421 14809 hybrid_clock.cc:648] HybridClock initialized: now 1790516752804377 us; error 43 us; skew 500 ppm
I20260927 13:45:52.804714 14809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:52 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516752, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:52.807636 14809 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:45:52.809036 14809 webserver.cc:460] Webserver started at http://127.1.237.254:32827/ using document root <none> and password file <none>
I20260927 13:45:52.809746 14809 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:52.809886 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:45:52 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516752, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:52.880812 14809 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 13:45:52.884352 14825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:52.886018 14809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:45:52.886219 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "5c605b19363a4a97aa79e265ec4ed192"
format_stamp: "Formatted at 2026-09-27 13:45:49 on dist-test-slave-7g3l"
server_key: "8be24317b5b76e2089fe6e5d16446856"
server_key_iv: "df5709cda8f646638cdb5856615f7bff"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:52.887079 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:52.906293 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:52.912514 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:52.912820 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:52.935910 14809 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:38635
I20260927 13:45:52.935956 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:38635 every 8 connection(s)
I20260927 13:45:52.937901 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:45:52.941687 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14809
I20260927 13:45:52.942866 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:38731
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:38635
--webserver_port=37693
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:45:52.952294 14877 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:45:52.957552 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Bootstrap starting.
I20260927 13:45:52.984455 14877 log.cc:824] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:52.994755 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:45:52.995358 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Bootstrap complete.
I20260927 13:45:53.008610 14877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:53.010090 14877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c605b19363a4a97aa79e265ec4ed192, State: Initialized, Role: FOLLOWER
I20260927 13:45:53.010936 14877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:53.011267 14877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:45:53.011428 14877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:45:53.011652 14877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:45:53.017318 14877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:53.017928 14877 leader_election.cc:302] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c605b19363a4a97aa79e265ec4ed192; no voters:
I20260927 13:45:53.018563 14877 leader_election.cc:288] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:45:53.018692 14881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:45:53.019867 14881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [term 2 LEADER]: Becoming Leader. State: Replica: 5c605b19363a4a97aa79e265ec4ed192, State: Running, Role: LEADER
I20260927 13:45:53.020476 14877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:45:53.020394 14881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } }
I20260927 13:45:53.022526 14883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c605b19363a4a97aa79e265ec4ed192. Latest consensus state: current_term: 2 leader_uuid: "5c605b19363a4a97aa79e265ec4ed192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } } }
I20260927 13:45:53.022881 14883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:53.023434 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c605b19363a4a97aa79e265ec4ed192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c605b19363a4a97aa79e265ec4ed192" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38635 } } }
I20260927 13:45:53.023756 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192 [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:53.026166 14887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:45:53.031658 14887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:45:53.032948 14887 catalog_manager.cc:1259] Loaded cluster ID: 47904ee51e2b402baf0f037a05dc1c46
I20260927 13:45:53.033075 14887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:45:53.037019 14887 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:45:53.038337 14887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Loaded TSK: 0
I20260927 13:45:53.039794 14887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:45:53.042555 14877 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 13:45:53.369910 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:53.370299 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:53.370512 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:53.381735 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:53.381894 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:53.382118 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:45:53.397027 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:38731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=37693
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:53.398780 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:53.400521 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:53.430269 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:53.431247 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:53.440760 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:53.442447 14879 server_base.cc:1034] running on GCE node
I20260927 13:45:53.443181 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:53.447340 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:53.448581 14879 hybrid_clock.cc:648] HybridClock initialized: now 1790516753448474 us; error 92 us; skew 500 ppm
I20260927 13:45:53.448951 14879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:53 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516753, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:53.452972 14879 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:45:53.460572 14879 webserver.cc:460] Webserver started at http://127.1.237.193:37693/ using document root <none> and password file <none>
I20260927 13:45:53.461246 14879 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:53.461405 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:45:53 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516753, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:53.504858 14879 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260927 13:45:53.507720 14926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:53.509155 14879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 13:45:53.509330 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "6875036f044a475bb1c247f6b3c8d026"
format_stamp: "Formatted at 2026-09-27 13:45:50 on dist-test-slave-7g3l"
server_key: "82f5d32396c5868e6812929f0c82ee3e"
server_key_iv: "94414f0b1a654e8fcfee48f27628ae74"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:53.510414 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:53.555290 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:53.559573 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:53.559916 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:53.561038 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:53.563823 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:53.563944 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:53.564054 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:53.564139 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:53.621116 14879 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:38731
I20260927 13:45:53.621205 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:38731 every 8 connection(s)
I20260927 13:45:53.622913 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:45:53.629607 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 14879
I20260927 13:45:53.630407 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:36693
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:38635
--webserver_port=43921
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:53 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516753, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:53.652945 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:53.627504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:43411 (local address 127.1.237.254:38635)
0927 13:45:53.629760 (+ 2256us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:53.629783 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:53.632316 (+ 2533us) server_negotiation.cc:407] Connection header received
0927 13:45:53.635309 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:53.635330 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:53.635761 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:53.636034 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:53.637826 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:53.639117 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:53.640974 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:53.641580 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:53.645652 (+ 4072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:53.645889 (+ 237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:53.645895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:53.645907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:53.645937 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:53.648671 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:53.649417 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:53.649428 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:53.649433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:53.649581 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:53.649997 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:53.650005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:53.650011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:53.650435 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:53.650620 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:53.651386 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 13:45:53.651741 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1915,"thread_start_us":8435,"threads_started":1}
I20260927 13:45:53.652961 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:53.626400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.193:43411)
0927 13:45:53.630408 (+ 4008us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:53.630472 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:53.633907 (+ 3435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:53.636588 (+ 2681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:53.636595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:53.637133 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:53.637641 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:53.637657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:53.639369 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:53.639374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:53.640808 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:53.640819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:53.641821 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:53.641829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:53.642109 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:53.642896 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:53.642930 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:53.645383 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:53.648809 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:53.648818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:53.648831 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:53.649280 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:53.649699 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:53.649704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:53.649707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:53.649887 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:53.650574 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:53.650582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:53.650873 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 13:45:53.651152 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 13:45:53.651675 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3662,"thread_start_us":8850,"threads_started":1}
I20260927 13:45:53.656229 15040 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:53.656554 15040 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:53.657337 15040 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
W20260927 13:45:53.660468 15040 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:38635 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:45:53.660727 15040 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:53.661023 15040 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:53.662134 15040 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:53.662372 15040 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
W20260927 13:45:53.663380 15040 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:45:54.034317 15044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:54.034606 15044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:54.034770 15044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:54.041949 15044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:54.042086 15044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:54.042249 15044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:45:54.051884 15044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:36693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=43921
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:54.053121 15044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:54.054438 15044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:54.068374 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:54.068651 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:54.069145 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:54.069437 15044 server_base.cc:1034] running on GCE node
I20260927 13:45:54.069963 15044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:54.070834 15044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:54.072079 15044 hybrid_clock.cc:648] HybridClock initialized: now 1790516754072035 us; error 25 us; skew 500 ppm
I20260927 13:45:54.072386 15044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:54.079242 15044 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:45:54.080938 15044 webserver.cc:460] Webserver started at http://127.1.237.194:43921/ using document root <none> and password file <none>
I20260927 13:45:54.081555 15044 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:54.081698 15044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:54.106552 15044 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 13:45:54.109858 15067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:54.111286 15044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:45:54.111487 15044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "21bc0d94362c422a93928f5919382762"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "653a6fcc5dc84729839876e923dfb97a"
server_key_iv: "ee3fdc4fc8680e8feb0fc77a76367f75"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:54.112540 15044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:54.131695 15044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:54.136970 15044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:54.137275 15044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:54.138407 15044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:54.140084 15044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:54.140190 15044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:54.140285 15044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:54.140368 15044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:54.178543 15044 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:36693
I20260927 13:45:54.178560 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:36693 every 8 connection(s)
I20260927 13:45:54.180258 15044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:45:54.184024 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15044
I20260927 13:45:54.184708 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.237.212:50523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:34429
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:38635
--webserver_port=42319
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:54.205358 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:54.182674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:47963 (local address 127.1.237.254:38635)
0927 13:45:54.183118 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:54.183124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:54.187915 (+ 4791us) server_negotiation.cc:407] Connection header received
0927 13:45:54.190653 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:54.190660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:54.190747 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:54.190832 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:54.193152 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.194270 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.196106 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.196507 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.196639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:54.199871 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:54.199879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:54.199887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:54.199917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:54.202488 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.203174 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.203181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.203187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.203306 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.203694 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.203698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.203702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.203969 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:54.204124 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:54.204774 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 13:45:54.204992 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260927 13:45:54.206473 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:54.183313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.194:47963)
0927 13:45:54.187719 (+ 4406us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:54.187765 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:54.190159 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:54.191517 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:54.191529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:54.192151 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:54.192934 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:54.192958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.194492 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.194498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:54.195965 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:54.195976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.196718 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.196726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:54.196958 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:54.197645 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:54.197667 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:54.199639 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:54.202618 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:54.202626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:54.202642 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:54.203028 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:54.203407 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:54.203412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:54.203416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:54.203582 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:54.204089 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:54.204098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:54.204408 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 13:45:54.204671 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 13:45:54.205196 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4111,"thread_start_us":8008,"threads_started":1}
I20260927 13:45:54.208047 15182 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:54.208364 15182 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:54.209074 15182 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:54.210726 14842 ts_manager.cc:194] Registered new tserver with Master: 21bc0d94362c422a93928f5919382762 (127.1.237.194:36693)
I20260927 13:45:54.212841 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:47963
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:45:54.559636 15186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:54.559973 15186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:54.560115 15186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:54.567716 15186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:54.567857 15186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:54.568050 15186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:45:54.582540 15186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.237.212:50523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=42319
--tserver_master_addrs=127.1.237.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:54.584275 15186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:54.586004 15186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:54.610348 15195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:54.610492 15194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:54.615453 15197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:54.620270 15186 server_base.cc:1034] running on GCE node
I20260927 13:45:54.621160 15186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:54.629864 15186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:54.631081 15186 hybrid_clock.cc:648] HybridClock initialized: now 1790516754631020 us; error 56 us; skew 500 ppm
I20260927 13:45:54.631461 15186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:54.636803 15186 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:45:54.642050 15186 webserver.cc:460] Webserver started at http://127.1.237.195:42319/ using document root <none> and password file <none>
I20260927 13:45:54.642918 15186 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:54.643059 15186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:54.663517 15040 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:54.663920 15040 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:54.665143 14842 ts_manager.cc:194] Registered new tserver with Master: 6875036f044a475bb1c247f6b3c8d026 (127.1.237.193:38731)
I20260927 13:45:54.666275 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:43411
I20260927 13:45:54.690924 15186 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.007s sys 0.006s
I20260927 13:45:54.694783 15205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:54.696640 15186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:45:54.696849 15186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c37616d4061b4d3fa8323d9cfb849ef9"
format_stamp: "Formatted at 2026-09-27 13:45:51 on dist-test-slave-7g3l"
server_key: "23ca0a83a3357c2f24b9cf4ece13238d"
server_key_iv: "02504de376f2ae523388584a0f9620cb"
server_key_version: "kuduclusterkey@0"
I20260927 13:45:54.698112 15186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:54.730762 15186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:54.735598 15186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:54.735869 15186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:54.736935 15186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:45:54.740064 15186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:45:54.740160 15186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:54.740247 15186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:45:54.740314 15186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:54.908914 15186 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34429
I20260927 13:45:54.908968 15317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34429 every 8 connection(s)
I20260927 13:45:54.910329 15186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:45:54.919961 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15186
Sep 27 13:45:54 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:54.988531 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:54.921624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.1.237.254:38635)
0927 13:45:54.962611 (+ 40987us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:54.962618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:54.962642 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:45:54.962738 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:54.962746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:54.962863 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:54.963001 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:54.965782 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.971245 (+ 5463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.974954 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.975935 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.976102 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:54.981158 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:54.981166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:54.981173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:54.981212 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:54.984795 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.986062 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.986068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.986071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.986205 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.986626 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.986631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.986635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.986998 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:54.987528 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:54.987768 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 13:45:54.987999 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40858,"thread_start_us":39666,"threads_started":1}
I20260927 13:45:54.995365 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:54.914946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:36645 (local address 127.1.237.254:38635)
0927 13:45:54.961272 (+ 46326us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:54.961281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:54.971204 (+ 9923us) server_negotiation.cc:407] Connection header received
0927 13:45:54.977163 (+ 5959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:54.977167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:54.977251 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:54.977361 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:54.979646 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.980910 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.983433 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.984136 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.984251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:54.989044 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:54.989051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:54.989055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:54.989086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:54.992468 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.993157 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.993162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.993166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.993278 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:54.993720 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:54.993724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:54.993728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:54.993968 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:54.994114 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:54.994773 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 13:45:54.994971 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46170,"thread_start_us":48210,"threads_started":1}
I20260927 13:45:54.996654 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:54.914234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38635 (local address 127.1.237.195:36645)
0927 13:45:54.967925 (+ 53691us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:54.967991 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:54.976923 (+ 8932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:54.977791 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:54.977804 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:54.978489 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:54.979429 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:54.979458 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.981098 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.981103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:54.983245 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:54.983262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:54.984655 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:54.984665 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:54.984948 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:54.985661 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:54.985692 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:54.988779 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:54.992610 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:54.992619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:54.992636 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:54.993033 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:54.993387 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:54.993392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:54.993396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:54.993597 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:54.994102 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:54.994110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:54.994426 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 13:45:54.994674 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 13:45:54.995248 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53246,"thread_start_us":57034,"threads_started":1}
I20260927 13:45:54.998029 15318 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38635
I20260927 13:45:54.998333 15318 heartbeater.cc:464] Registering TS with master...
I20260927 13:45:54.999004 15318 heartbeater.cc:511] Master 127.1.237.254:38635 requested a full tablet report, sending...
I20260927 13:45:55.000469 14842 ts_manager.cc:194] Registered new tserver with Master: c37616d4061b4d3fa8323d9cfb849ef9 (127.1.237.195:34429)
I20260927 13:45:55.001398 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:36645
I20260927 13:45:55.004500 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:45:55.029134 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.007789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.1.237.254:38635)
0927 13:45:55.007956 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.007961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.013586 (+ 5625us) server_negotiation.cc:407] Connection header received
0927 13:45:55.013749 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.013755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.013834 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.013946 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:55.015829 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.016916 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.018663 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.019229 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.019361 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.022643 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:55.022649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:55.022653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:55.022692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:55.025967 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.026785 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.026791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.026794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.026968 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.027431 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.027439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.027443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.027800 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:55.028077 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.028553 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.028788 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:45:55.034832 14842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:45:55.045473 14842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 13:45:55 dist-test-slave-7g3l krb5kdc[13832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516754, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for HTTP/127.1.237.212@KRBTEST.COM
I20260927 13:45:55.105522 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.071776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34429 (local address 127.0.0.1:48434)
0927 13:45:55.085926 (+ 14150us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.085967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:55.086132 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:55.096474 (+ 10342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:55.096482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:55.096532 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:55.096933 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.096947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.098944 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.098951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.102469 (+ 3518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.102484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.104473 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.104522 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.104735 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.104814 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 13:45:55.104938 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 13:45:55.105140 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13871,"thread_start_us":14599,"threads_started":1}
I20260927 13:45:55.106907 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.072415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.1.237.195:34429)
0927 13:45:55.093579 (+ 21164us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.093592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.093633 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:45:55.093752 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.093762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.094102 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.096440 (+ 2338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:55.097152 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.098758 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.102669 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.104210 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.106082 (+ 1872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.106322 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.106394 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.106562 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20934,"thread_start_us":20753,"threads_started":1}
I20260927 13:45:55.113235 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.075959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:36693 (local address 127.0.0.1:39744)
0927 13:45:55.086110 (+ 10151us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.086134 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:55.086281 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:55.100306 (+ 14025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:55.100310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:55.100337 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:55.100680 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.100689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.103664 (+ 2975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.103669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.105544 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.105558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.112431 (+ 6873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.112437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.112618 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.112649 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:45:55.112724 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:45:55.112862 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9983,"mutex_wait_us":1,"thread_start_us":9520,"threads_started":1}
I20260927 13:45:55.114014 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.080660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.1.237.194:36693)
0927 13:45:55.097702 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.097714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.097747 (+ 33us) server_negotiation.cc:407] Connection header received
0927 13:45:55.097851 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.097862 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.098139 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.098354 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:55.101975 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.103441 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.110460 (+ 7019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.111929 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.112034 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.112218 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.113557 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.113714 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16820,"thread_start_us":17855,"threads_started":1}
I20260927 13:45:55.116390 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.093130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38154)
0927 13:45:55.105810 (+ 12680us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.105827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:55.105926 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:55.110060 (+ 4134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:55.110065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:55.110089 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:55.110227 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.110235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.112013 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.112019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.113860 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.113882 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.115709 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.115727 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.115936 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.115966 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:45:55.116027 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 13:45:55.116136 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12563,"thread_start_us":14016,"threads_started":1}
I20260927 13:45:55.117110 15253 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d (DEFAULT_TABLE table=test-table [id=9dd58b86777c40beb6ccab104862b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:55.118052 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.087238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.1.237.193:38731)
0927 13:45:55.109451 (+ 22213us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.109460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.109488 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:45:55.109592 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.109599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.109853 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.110052 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:55.110420 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.111863 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.114079 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.115502 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.117179 (+ 1677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.117646 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.117697 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.117805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22031,"thread_start_us":22969,"threads_started":1}
I20260927 13:45:55.118561 15253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcb1d3a611a44d58f46885ce2aa686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:55.118068 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d (DEFAULT_TABLE table=test-table [id=9dd58b86777c40beb6ccab104862b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:55.119375 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcb1d3a611a44d58f46885ce2aa686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:55.126492 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d (DEFAULT_TABLE table=test-table [id=9dd58b86777c40beb6ccab104862b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:45:55.128244 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcb1d3a611a44d58f46885ce2aa686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:55.151249 15340 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Bootstrap starting.
I20260927 13:45:55.157052 15340 tablet_bootstrap.cc:654] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:55.158358 15340 log.cc:824] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:55.159832 15342 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Bootstrap starting.
I20260927 13:45:55.163228 15342 tablet_bootstrap.cc:654] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:55.163740 15341 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Bootstrap starting.
I20260927 13:45:55.164724 15342 log.cc:824] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:55.166949 15341 tablet_bootstrap.cc:654] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:55.168155 15341 log.cc:824] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:55.171274 15340 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: No bootstrap required, opened a new log
I20260927 13:45:55.171598 15340 ts_tablet_manager.cc:1397] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.003s
I20260927 13:45:55.173660 15342 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: No bootstrap required, opened a new log
I20260927 13:45:55.173911 15342 ts_tablet_manager.cc:1397] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260927 13:45:55.183219 15340 raft_consensus.cc:348] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.183820 15340 raft_consensus.cc:374] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:55.183971 15340 raft_consensus.cc:729] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37616d4061b4d3fa8323d9cfb849ef9, State: Initialized, Role: FOLLOWER
I20260927 13:45:55.184712 15340 consensus_queue.cc:260] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.195461 15342 raft_consensus.cc:348] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.195946 15342 raft_consensus.cc:374] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:55.196053 15342 raft_consensus.cc:729] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bc0d94362c422a93928f5919382762, State: Initialized, Role: FOLLOWER
I20260927 13:45:55.200464 15342 consensus_queue.cc:260] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.204416 15318 heartbeater.cc:503] Master 127.1.237.254:38635 was elected leader, sending a full tablet report...
I20260927 13:45:55.204973 15340 ts_tablet_manager.cc:1428] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Time spent starting tablet: real 0.033s user 0.011s sys 0.017s
I20260927 13:45:55.207654 15342 ts_tablet_manager.cc:1428] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Time spent starting tablet: real 0.034s user 0.014s sys 0.002s
I20260927 13:45:55.208005 15182 heartbeater.cc:503] Master 127.1.237.254:38635 was elected leader, sending a full tablet report...
I20260927 13:45:55.215720 15341 tablet_bootstrap.cc:492] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: No bootstrap required, opened a new log
I20260927 13:45:55.216032 15341 ts_tablet_manager.cc:1397] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20260927 13:45:55.223138 15346 raft_consensus.cc:482] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:45:55.223424 15346 raft_consensus.cc:504] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.225312 15346 leader_election.cc:288] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6875036f044a475bb1c247f6b3c8d026 (127.1.237.193:38731), 21bc0d94362c422a93928f5919382762 (127.1.237.194:36693)
I20260927 13:45:55.226807 15341 raft_consensus.cc:348] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.227255 15341 raft_consensus.cc:374] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:55.227340 15341 raft_consensus.cc:729] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6875036f044a475bb1c247f6b3c8d026, State: Initialized, Role: FOLLOWER
W20260927 13:45:55.228281 15319 tablet.cc:2369] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:45:55.228199 15341 consensus_queue.cc:260] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.234153 15341 ts_tablet_manager.cc:1428] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Time spent starting tablet: real 0.018s user 0.008s sys 0.007s
I20260927 13:45:55.234552 15040 heartbeater.cc:503] Master 127.1.237.254:38635 was elected leader, sending a full tablet report...
I20260927 13:45:55.235694 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.225747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.1.237.195:43885)
0927 13:45:55.226035 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.226055 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:55.226160 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:55.227702 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:55.227707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:55.227733 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:55.227945 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.227957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.230389 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.230394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.233203 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.233216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.234977 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.234983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.235178 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.235225 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 13:45:55.235310 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 13:45:55.235441 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260927 13:45:55.236501 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.225809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:43885 (local address 127.1.237.193:38731)
0927 13:45:55.226530 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.226536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.226554 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:45:55.226617 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.226621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.226686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.226773 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:55.229095 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.230227 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.233383 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.234641 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.234803 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.234882 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.236146 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.236289 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260927 13:45:55.237279 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcb1d3a611a44d58f46885ce2aa686d" candidate_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6875036f044a475bb1c247f6b3c8d026" is_pre_election: true
I20260927 13:45:55.237756 14994 raft_consensus.cc:2454] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37616d4061b4d3fa8323d9cfb849ef9 in term 0.
I20260927 13:45:55.238606 15208 leader_election.cc:302] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6875036f044a475bb1c247f6b3c8d026, c37616d4061b4d3fa8323d9cfb849ef9; no voters:
I20260927 13:45:55.239004 15346 raft_consensus.cc:2781] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:45:55.239142 15346 raft_consensus.cc:482] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:45:55.239264 15346 raft_consensus.cc:3037] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:55.242219 15346 raft_consensus.cc:504] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.243886 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcb1d3a611a44d58f46885ce2aa686d" candidate_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6875036f044a475bb1c247f6b3c8d026"
I20260927 13:45:55.244084 14994 raft_consensus.cc:3037] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:55.245767 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.227555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:45901 (local address 127.1.237.194:36693)
0927 13:45:55.227781 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.227787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.229177 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 13:45:55.229415 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.229424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.229523 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.229632 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:45:55.230144 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.231532 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.243452 (+ 11920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.244669 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.244785 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.244856 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.245398 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.245530 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 13:45:55.246414 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.227112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:36693 (local address 127.1.237.195:45901)
0927 13:45:55.229047 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.229067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:55.229269 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:55.229645 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:55.229650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:55.229673 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:45:55.229977 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.229985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.231778 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.231786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.243249 (+ 11463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:55.243275 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.245042 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.245049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:55.245210 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.245241 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:45:55.245936 (+ 695us) client_negotiation.cc:240] Negotiation successful
0927 13:45:55.246068 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":1906,"thread_start_us":7404,"threads_started":1}
I20260927 13:45:55.247058 14994 raft_consensus.cc:2454] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37616d4061b4d3fa8323d9cfb849ef9 in term 1.
I20260927 13:45:55.247395 15137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcb1d3a611a44d58f46885ce2aa686d" candidate_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bc0d94362c422a93928f5919382762" is_pre_election: true
I20260927 13:45:55.247870 15208 leader_election.cc:302] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6875036f044a475bb1c247f6b3c8d026, c37616d4061b4d3fa8323d9cfb849ef9; no voters:
I20260927 13:45:55.247933 15137 raft_consensus.cc:2454] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37616d4061b4d3fa8323d9cfb849ef9 in term 0.
I20260927 13:45:55.249444 15346 leader_election.cc:288] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [CANDIDATE]: Term 1 election: Requested vote from peers 6875036f044a475bb1c247f6b3c8d026 (127.1.237.193:38731), 21bc0d94362c422a93928f5919382762 (127.1.237.194:36693)
I20260927 13:45:55.249640 15346 raft_consensus.cc:2781] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:45:55.249775 15346 raft_consensus.cc:686] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 1 LEADER]: Becoming Leader. State: Replica: c37616d4061b4d3fa8323d9cfb849ef9, State: Running, Role: LEADER
I20260927 13:45:55.249971 15137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcb1d3a611a44d58f46885ce2aa686d" candidate_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bc0d94362c422a93928f5919382762"
I20260927 13:45:55.250202 15137 raft_consensus.cc:3037] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:55.250281 15346 consensus_queue.cc:237] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } }
I20260927 13:45:55.254148 15137 raft_consensus.cc:2454] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37616d4061b4d3fa8323d9cfb849ef9 in term 1.
I20260927 13:45:55.264071 14841 catalog_manager.cc:5495] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37616d4061b4d3fa8323d9cfb849ef9 (127.1.237.195). New cstate: current_term: 1 leader_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37616d4061b4d3fa8323d9cfb849ef9" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 } health_report { overall_health: UNKNOWN } } }
I20260927 13:45:55.289134 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.281716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.1.237.195:34429)
0927 13:45:55.281897 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.281902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.281923 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:45:55.282170 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.282176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.282269 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.282422 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:45:55.282975 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.284053 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.285960 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.286590 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.286705 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.287371 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:45:55.288015 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:45:55.288164 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.288799 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.288878 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:45:55.300324 15137 raft_consensus.cc:1261] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 1 FOLLOWER]: Refusing update from remote peer c37616d4061b4d3fa8323d9cfb849ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:45:55.301376 15350 consensus_queue.cc:1035] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bc0d94362c422a93928f5919382762" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:45:55.309074 14994 raft_consensus.cc:1261] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 1 FOLLOWER]: Refusing update from remote peer c37616d4061b4d3fa8323d9cfb849ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:45:55.310150 15350 consensus_queue.cc:1035] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6875036f044a475bb1c247f6b3c8d026" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:45:55.351472 14841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:45:55.351799 14841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:45:55.355988 14841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c605b19363a4a97aa79e265ec4ed192: Sending DeleteTablet for 3 replicas of tablet 1dcb1d3a611a44d58f46885ce2aa686d
I20260927 13:45:55.357024 15253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:55 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:48434
I20260927 13:45:55.358808 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:55 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:38154
I20260927 13:45:55.366487 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dcb1d3a611a44d58f46885ce2aa686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:45:55 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:39744
I20260927 13:45:55.371989 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14879
W20260927 13:45:55.386231 15041 tablet.cc:2369] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:45:55.399930 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.374082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.1.237.193:38731)
0927 13:45:55.374253 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.374258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.374275 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:45:55.374331 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.374337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.374425 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.374522 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:55.381674 (+ 7152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.382697 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.387694 (+ 4997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.388306 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.388434 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.392168 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:55.392182 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:55.392200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:55.392238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:55.396070 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.396948 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.396958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.396962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.397115 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.397547 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.397552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.397556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.397972 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:55.398244 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.399308 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.399547 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:45:55.400630 15035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:55.414274 15363 tablet_replica.cc:330] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: stopping tablet replica
I20260927 13:45:55.414784 15363 raft_consensus.cc:2229] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:45:55.415350 15363 raft_consensus.cc:2258] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:45:55.419274 15361 tablet_replica.cc:330] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: stopping tablet replica
I20260927 13:45:55.419708 15361 raft_consensus.cc:2229] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:45:55.420485 15361 raft_consensus.cc:2258] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:45:55.424249 15363 ts_tablet_manager.cc:1905] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:45:55.429939 15361 ts_tablet_manager.cc:1905] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:45:55.444496 15361 ts_tablet_manager.cc:1918] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:45:55.444651 15361 log.cc:1196] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/1dcb1d3a611a44d58f46885ce2aa686d
I20260927 13:45:55.445170 15361 ts_tablet_manager.cc:1939] T 1dcb1d3a611a44d58f46885ce2aa686d P c37616d4061b4d3fa8323d9cfb849ef9: Deleting consensus metadata
I20260927 13:45:55.446909 14828 catalog_manager.cc:4841] TS c37616d4061b4d3fa8323d9cfb849ef9 (127.1.237.195:34429): tablet 1dcb1d3a611a44d58f46885ce2aa686d (table test-table [id=9dd58b86777c40beb6ccab104862b0dd]) successfully deleted
I20260927 13:45:55.448542 15363 ts_tablet_manager.cc:1918] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:45:55.448693 15363 log.cc:1196] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/1dcb1d3a611a44d58f46885ce2aa686d
I20260927 13:45:55.449170 15363 ts_tablet_manager.cc:1939] T 1dcb1d3a611a44d58f46885ce2aa686d P 21bc0d94362c422a93928f5919382762: Deleting consensus metadata
I20260927 13:45:55.450837 14829 catalog_manager.cc:4841] TS 21bc0d94362c422a93928f5919382762 (127.1.237.194:36693): tablet 1dcb1d3a611a44d58f46885ce2aa686d (table test-table [id=9dd58b86777c40beb6ccab104862b0dd]) successfully deleted
I20260927 13:45:55.594116 15362 tablet_replica.cc:330] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026: stopping tablet replica
I20260927 13:45:55.596374 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14879
I20260927 13:45:55.596449 15362 raft_consensus.cc:2229] T 1dcb1d3a611a44d58f46885ce2aa686d P 6875036f044a475bb1c247f6b3c8d026 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 13:45:55.619428 14828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:38731 (error 108)
I20260927 13:45:55.619620 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15044
W20260927 13:45:55.620366 14826 connection.cc:535] server connection from 127.1.237.193:43411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 13:45:55.636484 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.621405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38176)
0927 13:45:55.636146 (+ 14741us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.636279 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14625,"threads_started":1}
W20260927 13:45:55.636884 14828 catalog_manager.cc:4584] TS 6875036f044a475bb1c247f6b3c8d026 (127.1.237.193:38731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1dcb1d3a611a44d58f46885ce2aa686d: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
I20260927 13:45:55.643460 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.621274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.1.237.194:36693)
0927 13:45:55.621436 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.621442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.621459 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:45:55.621536 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.621541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.621615 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.621708 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:55.623708 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.624810 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.626892 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.627698 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.627823 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.636132 (+ 8309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:55.636150 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:55.636168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:55.636209 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:55.640235 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.641029 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.641038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.641042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.641178 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.641613 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.641618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.641621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.642087 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:55.642381 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.642688 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.642932 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:45:55.644191 15177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:55.690591 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.690018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38188)
0927 13:45:55.690307 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.690429 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 13:45:55.692024 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.691547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38192)
0927 13:45:55.691778 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.691861 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 13:45:55.752792 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.752176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38200)
0927 13:45:55.752494 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.752615 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260927 13:45:55.754141 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.753681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38212)
0927 13:45:55.753878 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.753962 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 13:45:55.787894 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15044
I20260927 13:45:55.811558 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15186
I20260927 13:45:55.832494 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.812402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.1.237.195:34429)
0927 13:45:55.815851 (+ 3449us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:55.815857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:55.815877 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:45:55.815943 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:55.815948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:55.816025 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:55.816136 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:55.817816 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.818994 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.820931 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:55.821677 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:55.821803 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:55.825150 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:55.825175 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:55.825196 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:55.825240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:55.829348 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.830159 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.830167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.830171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.830366 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:55.830760 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:55.830765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:55.830768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:55.831215 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:55.831510 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:55.831837 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 13:45:55.832030 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3315,"thread_start_us":3089,"threads_started":1}
I20260927 13:45:55.833460 15313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:55.834837 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.834353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38214)
0927 13:45:55.834552 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.834652 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:45:55.835973 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:55.835622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38220)
0927 13:45:55.835810 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:55.835867 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 13:45:55.959372 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15186
I20260927 13:45:55.980100 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 14809
I20260927 13:45:55.981199 14872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:45:56.089187 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:56.088652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:38731 (local address 127.0.0.1:38236)
0927 13:45:56.088893 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:56.088979 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:38731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 13:45:56.089421 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 14809
I20260927 13:45:56.113220 1975 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 13:45:56.447537 1975 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 13:45:56.447767 1975 mini_ranger.cc:66] Stopping Ranger...
I20260927 13:45:56.792172 1975 mini_ranger.cc:68] Stopped Ranger
2026-09-27 13:45:56.792 UTC [13862] LOG: received smart shutdown request
2026-09-27 13:45:56.795 UTC [13862] LOG: background worker "logical replication launcher" (PID 13871) exited with exit code 1
2026-09-27 13:45:56.795 UTC [13866] LOG: shutting down
2026-09-27 13:45:56.940 UTC [13862] LOG: database system is shut down
2026-09-27T13:45:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50655 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:45:57 dist-test-slave-7g3l krb5kdc[15387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:45:57 dist-test-slave-7g3l krb5kdc[15387](info): set up 2 sockets
Sep 27 13:45:57 dist-test-slave-7g3l krb5kdc[15387](info): commencing operation
krb5kdc: starting...
W20260927 13:45:58.151543 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:45:58 dist-test-slave-7g3l krb5kdc[15387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:45:58.366726 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41653
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:45:58.679426 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:58.679731 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:58.679909 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:58.686808 15400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:58.686944 15400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:58.687043 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:58.687142 15400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:58.687238 15400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:58.696471 15400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:58.697707 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:58.698911 15400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:58.710644 15405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:58.710596 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:58.710681 15406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:58.711426 15400 server_base.cc:1034] running on GCE node
I20260927 13:45:58.712106 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:58.713371 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:58.714560 15400 hybrid_clock.cc:648] HybridClock initialized: now 1790516758714508 us; error 52 us; skew 500 ppm
I20260927 13:45:58.714855 15400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:58 dist-test-slave-7g3l krb5kdc[15387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:58.717921 15400 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:45:58.719283 15400 webserver.cc:460] Webserver started at http://127.1.237.254:37385/ using document root <none> and password file <none>
I20260927 13:45:58.719933 15400 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:58.720060 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:58.720373 15400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:45:58.723187 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "55fa6348464042cab82b66168e51302b"
format_stamp: "Formatted at 2026-09-27 13:45:58 on dist-test-slave-7g3l"
server_key: "0b29999477c5c836cbc3859d65d11148"
server_key_iv: "bb4952edfba0927a9bebd7b401d8afcc"
server_key_version: "encryptionkey@0"
I20260927 13:45:58.724009 15400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "55fa6348464042cab82b66168e51302b"
format_stamp: "Formatted at 2026-09-27 13:45:58 on dist-test-slave-7g3l"
server_key: "0b29999477c5c836cbc3859d65d11148"
server_key_iv: "bb4952edfba0927a9bebd7b401d8afcc"
server_key_version: "encryptionkey@0"
I20260927 13:45:58.728984 15400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 13:45:58.732148 15415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:58.733458 15400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:45:58.733626 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "55fa6348464042cab82b66168e51302b"
format_stamp: "Formatted at 2026-09-27 13:45:58 on dist-test-slave-7g3l"
server_key: "0b29999477c5c836cbc3859d65d11148"
server_key_iv: "bb4952edfba0927a9bebd7b401d8afcc"
server_key_version: "encryptionkey@0"
I20260927 13:45:58.733812 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:58.743767 15400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:58.747243 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:58.747509 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:58.764811 15400 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:41653
I20260927 13:45:58.764885 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:41653 every 8 connection(s)
I20260927 13:45:58.766076 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:45:58.769409 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:58.775367 15467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:45:58.776213 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15400
I20260927 13:45:58.776428 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:45:58.777016 1975 external_mini_cluster.cc:1428] Setting key 2103b3be5defe21ce1e9afb74ffb3b62
Sep 27 13:45:58 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:58.797863 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.778176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41653 (local address 127.0.0.1:41006)
0927 13:45:58.778917 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.778960 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:58.780900 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:58.781765 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:58.781769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:58.782262 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:58.782578 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:58.782594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:58.784554 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:58.784560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:58.785594 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:58.785605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:58.786498 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:58.786505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:58.786692 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:58.787366 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:58.787390 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:58.789675 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:58.794025 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:58.794037 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:58.794052 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:58.794464 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:58.794968 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:58.794978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:58.794980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:58.795124 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:58.796169 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:58.796175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:58.796359 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 13:45:58.796551 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 13:45:58.796883 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":339,"threads_started":1}
I20260927 13:45:58.797955 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.779384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.1.237.254:41653)
0927 13:45:58.780058 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:58.780069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:58.780121 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:45:58.781252 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:58.781262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:58.781524 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:58.781742 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:58.782772 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:58.784398 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:58.785734 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:58.786280 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:58.786469 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:58.789933 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:58.789945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:58.789961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:58.790006 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:58.793852 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:58.794600 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:58.794611 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:58.794617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:58.794842 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:58.795289 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:58.795298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:58.795303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:58.796028 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:58.796237 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:58.796761 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 13:45:58.797037 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.1.237.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.253@KRBTEST.COM" created.
I20260927 13:45:58.801102 15467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:45:58.802179 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.801705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53502)
0927 13:45:58.801903 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.802028 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:45:58.802438 15416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
I20260927 13:45:58.804047 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.803594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53514)
0927 13:45:58.803765 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.803865 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:45:58.804260 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:42623: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
I20260927 13:45:58.826552 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:42623
--webserver_interface=127.1.237.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:45:58.853605 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } attempt: 1
I20260927 13:45:58.854756 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.854308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53530)
0927 13:45:58.854500 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.854602 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 13:45:58.855767 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.855408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53534)
0927 13:45:58.855552 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.855622 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 13:45:58.855965 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:42623: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
I20260927 13:45:58.924185 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } attempt: 2
I20260927 13:45:58.925267 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.924855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53536)
0927 13:45:58.925036 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.925132 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 13:45:58.926374 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:58.926038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53552)
0927 13:45:58.926177 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:58.926246 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:45:58.926556 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:42623: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
I20260927 13:45:59.017792 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } attempt: 3
I20260927 13:45:59.019004 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.018578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53556)
0927 13:45:59.018768 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.018864 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 13:45:59.020035 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.019632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53570)
0927 13:45:59.019842 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.019917 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 13:45:59.020219 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:42623: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
W20260927 13:45:59.135514 15474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:59.135828 15474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:59.135987 15474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:59.142917 15474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:59.143052 15474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:59.143157 15474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:59.143267 15474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:59.143378 15474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:59.152401 15474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.237.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:59.153633 15474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:59.154868 15474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:45:59.163427 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } attempt: 4
I20260927 13:45:59.164796 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.164135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53580)
0927 13:45:59.164515 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.164615 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
W20260927 13:45:59.165634 15479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:59.166136 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.165558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53596)
0927 13:45:59.165918 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.166032 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
W20260927 13:45:59.166326 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:42623: Network error: Client connection negotiation failed: client connection to 127.1.237.253:42623: connect: Connection refused (error 111)
W20260927 13:45:59.166335 15480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:59.166819 15482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:59.167315 15474 server_base.cc:1034] running on GCE node
I20260927 13:45:59.167999 15474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:59.169222 15474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:59.170429 15474 hybrid_clock.cc:648] HybridClock initialized: now 1790516759170392 us; error 35 us; skew 500 ppm
I20260927 13:45:59.170775 15474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:59.181015 15474 init.cc:377] Logged in from keytab as kudu/127.1.237.253@KRBTEST.COM (short username kudu)
I20260927 13:45:59.182461 15474 webserver.cc:460] Webserver started at http://127.1.237.253:32957/ using document root <none> and password file <none>
I20260927 13:45:59.183140 15474 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:59.183329 15474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:59.183621 15474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:45:59.186519 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data/instance:
uuid: "122195df41a24e4d947c15969033716d"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "07fe3824b9e771c5bf5d62ea83f84a1a"
server_key_iv: "3251da96f49470380f5feeca32222459"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.187387 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal/instance:
uuid: "122195df41a24e4d947c15969033716d"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "07fe3824b9e771c5bf5d62ea83f84a1a"
server_key_iv: "3251da96f49470380f5feeca32222459"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.192250 15474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 13:45:59.195479 15489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:59.196739 15474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:45:59.196923 15474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "122195df41a24e4d947c15969033716d"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "07fe3824b9e771c5bf5d62ea83f84a1a"
server_key_iv: "3251da96f49470380f5feeca32222459"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.197119 15474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:59.228931 15474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:59.232842 15474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:59.233124 15474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:59.251442 15474 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.253:42623
I20260927 13:45:59.251474 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.253:42623 every 8 connection(s)
I20260927 13:45:59.252724 15474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
I20260927 13:45:59.255519 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15474
I20260927 13:45:59.255720 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-1/wal/instance
I20260927 13:45:59.256143 1975 external_mini_cluster.cc:1428] Setting key 2dd4120e93cd5bef957748c0a9d26030
I20260927 13:45:59.256417 15541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:59.262626 15541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.237.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.252@KRBTEST.COM" created.
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:59.283849 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.265172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.1.237.254:41653)
0927 13:45:59.265347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.265351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.266573 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 13:45:59.268408 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.268415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.268504 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.268610 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.269991 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.270670 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.272205 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.272633 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.272775 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.278098 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.278103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.278106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.278137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.280807 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.281496 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.281501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.281503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.281608 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.281998 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.282002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.282005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.282324 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.282488 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.283145 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.283422 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:45:59.284418 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.265716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41653 (local address 127.0.0.1:41010)
0927 13:45:59.266421 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.266456 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.268134 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.268830 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.268839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.269305 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.269826 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.269842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.270829 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.270834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.272064 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.272081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.272851 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.272859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.273064 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.273724 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.273746 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.277883 (+ 4137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.280946 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.280954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.280969 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.281371 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.281709 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.281714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.281718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.281887 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.282480 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.282486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.282769 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.282997 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.283406 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":293,"threads_started":1}
I20260927 13:45:59.286118 15541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:45:59.300372 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.286770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53612)
0927 13:45:59.286931 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.286957 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.287032 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.288018 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.288022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.288420 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.288539 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.288546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.289796 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.289803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.290575 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.290584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.291480 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.291486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.291692 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.292302 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.292324 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.295085 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.297900 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.297904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.297907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.298268 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.298697 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.298701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.298703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.298855 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.299557 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.299564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.299742 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.299871 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.300040 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 13:45:59.300875 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.287062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.1.237.253:42623)
0927 13:45:59.287656 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.287661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.287682 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:45:59.287717 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.287722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.287886 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.288029 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.288680 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.289668 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.290724 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.291324 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.291463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.295311 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.295321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.295333 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.295366 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.297788 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.298390 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.298396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.298398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.298532 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.298952 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.298954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.298956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.299378 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.299570 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.300143 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.300402 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
I20260927 13:45:59.303023 15541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:45:59.303583 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:44555
--webserver_interface=127.1.237.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:45:59.304195 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.303660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38330)
0927 13:45:59.303897 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.304042 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 13:45:59.304536 15493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.306157 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.305855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38342)
0927 13:45:59.305985 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.306065 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:45:59.306457 15541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.355822 15541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 1
I20260927 13:45:59.356998 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.356528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38344)
0927 13:45:59.356723 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.356850 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:45:59.358027 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.357695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38356)
0927 13:45:59.357865 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.357924 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:45:59.358275 15541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.426540 15541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 2
I20260927 13:45:59.427659 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.427212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38368)
0927 13:45:59.427414 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.427516 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 13:45:59.428574 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.428238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38382)
0927 13:45:59.428382 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.428453 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 13:45:59.428759 15541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.465590 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } attempt: 5
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:45:59.478145 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.466417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.1.237.253:42623)
0927 13:45:59.466613 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.466618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.466638 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:45:59.466778 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.466784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.466866 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.466973 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.467800 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.468492 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.469576 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.470062 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.470189 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.473172 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.473178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.473188 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.473217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.475734 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.476368 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.476375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.476378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.476508 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.476843 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.476849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.476852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.477116 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.477266 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.477621 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.477814 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 13:45:59.478145 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.466300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53624)
0927 13:45:59.466488 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.466511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.466634 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.467016 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.467022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.467453 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.467631 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.467640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.468615 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.468621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.469430 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.469449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.470211 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.470217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.470420 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.471041 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.471058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.472993 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.475897 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.475904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.475908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.476248 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.476613 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.476618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.476621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.476745 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.477234 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.477239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.477381 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.477536 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.477777 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:45:59.479111 15467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:45:59.480260 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.479846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38384)
0927 13:45:59.480025 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.480109 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:45:59.482244 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.481037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38396)
0927 13:45:59.482039 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.482128 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":879}
W20260927 13:45:59.482582 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.519966 15541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 3
I20260927 13:45:59.521056 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.520603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38410)
0927 13:45:59.520827 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.520912 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 13:45:59.522064 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.521688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38416)
0927 13:45:59.521902 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.521970 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260927 13:45:59.522296 15541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.533821 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 1
I20260927 13:45:59.534991 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.534455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38418)
0927 13:45:59.534665 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.534780 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:45:59.536090 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.535671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38428)
0927 13:45:59.535845 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.535935 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:45:59.536304 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.604547 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 2
I20260927 13:45:59.605799 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.605205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38434)
0927 13:45:59.605453 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.605586 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 13:45:59.606911 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.606482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38448)
0927 13:45:59.606667 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.606765 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:45:59.607131 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
W20260927 13:45:59.629560 15548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:45:59.629817 15548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:45:59.630000 15548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:45:59.637136 15548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:45:59.637226 15548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:45:59.637308 15548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:45:59.637409 15548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:45:59.637467 15548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:45:59.646724 15548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41653,127.1.237.253:42623,127.1.237.252:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.237.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:45:59.647986 15548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:45:59.649238 15548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:45:59.660768 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:59.660703 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:45:59.660804 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:45:59.661512 15548 server_base.cc:1034] running on GCE node
I20260927 13:45:59.662529 15548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:45:59.664465 15548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:45:59.665496 15541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 4
I20260927 13:45:59.665692 15548 hybrid_clock.cc:648] HybridClock initialized: now 1790516759665590 us; error 85 us; skew 500 ppm
I20260927 13:45:59.666106 15548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:45:59.666831 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.666157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38464)
0927 13:45:59.666567 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.666663 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260927 13:45:59.667868 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.667490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38466)
0927 13:45:59.667692 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.667771 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 13:45:59.668104 15541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:45:59.670351 15548 init.cc:377] Logged in from keytab as kudu/127.1.237.252@KRBTEST.COM (short username kudu)
I20260927 13:45:59.672185 15548 webserver.cc:460] Webserver started at http://127.1.237.252:32865/ using document root <none> and password file <none>
I20260927 13:45:59.673000 15548 fs_manager.cc:359] Metadata directory not provided
I20260927 13:45:59.673127 15548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:45:59.673517 15548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:45:59.677184 15548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data/instance:
uuid: "6a870c6729e643599673c4bc02ef5b90"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "ab15256c5d6b035cfb3a2cfe530a2a44"
server_key_iv: "e65ad9bb865b5a766e0a1290d0c4c096"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.677984 15548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal/instance:
uuid: "6a870c6729e643599673c4bc02ef5b90"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "ab15256c5d6b035cfb3a2cfe530a2a44"
server_key_iv: "e65ad9bb865b5a766e0a1290d0c4c096"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.682897 15548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:45:59.686106 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:45:59.687464 15548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:45:59.687631 15548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "6a870c6729e643599673c4bc02ef5b90"
format_stamp: "Formatted at 2026-09-27 13:45:59 on dist-test-slave-7g3l"
server_key: "ab15256c5d6b035cfb3a2cfe530a2a44"
server_key_iv: "e65ad9bb865b5a766e0a1290d0c4c096"
server_key_version: "encryptionkey@0"
I20260927 13:45:59.687810 15548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:45:59.707490 15548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:45:59.711647 15548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:45:59.711913 15548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:45:59.713380 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 3
I20260927 13:45:59.714466 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.714036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38476)
0927 13:45:59.714236 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.714321 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:45:59.715694 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.715119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38484)
0927 13:45:59.715518 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.715596 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
W20260927 13:45:59.715910 15467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:44555: Network error: Client connection negotiation failed: client connection to 127.1.237.252:44555: connect: Connection refused (error 111)
I20260927 13:45:59.729863 15548 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.252:44555
I20260927 13:45:59.729903 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.252:44555 every 8 connection(s)
I20260927 13:45:59.731113 15548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
I20260927 13:45:59.733630 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15548
I20260927 13:45:59.733850 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790516567150638-1975-0/minicluster-data/master-2/wal/instance
I20260927 13:45:59.734400 1975 external_mini_cluster.cc:1428] Setting key 813f0f4677412976d11006d47920006e
I20260927 13:45:59.734939 15615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:45:59.744534 15615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:45:59.758900 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.736244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.1.237.252:44555)
0927 13:45:59.737061 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.737075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.737121 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:45:59.739567 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.739595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.740104 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.740534 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.741814 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.743781 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.745068 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.746011 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.746195 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.750462 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.750472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.750494 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.750537 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.754233 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.755109 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.755117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.755126 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.755363 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.755837 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.755844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.755853 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.756572 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.756778 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.757296 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.757753 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":342,"threads_started":1}
I20260927 13:45:59.762494 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.747211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41653 (local address 127.0.0.1:41020)
0927 13:45:59.747906 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.747938 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.748131 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.748507 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.748514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.749048 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.749327 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.749335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.750439 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.750443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.751777 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.751793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.752924 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.752928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.753048 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.753673 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.753694 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.756350 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.759721 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.759730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.759736 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.760306 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.760728 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.760731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.760733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.760872 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.761470 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.761478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.761652 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.761813 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.762108 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":386,"threads_started":1}
I20260927 13:45:59.762555 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.747324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.1.237.254:41653)
0927 13:45:59.747547 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.747554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.748050 (+ 496us) server_negotiation.cc:407] Connection header received
0927 13:45:59.748263 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.748275 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.748336 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.748442 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.749473 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.750279 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.751969 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.752703 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.752852 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.756555 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.756560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.756563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.756592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.759601 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.760444 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.760452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.760456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.760620 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.761020 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.761024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.761027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.761347 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.761510 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.761945 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.762169 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:45:59.764389 15615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:45:59.778892 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.762236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.1.237.254:41653)
0927 13:45:59.762945 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.762954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.762974 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:45:59.763438 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.763442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.763495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.763615 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.765284 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.766163 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.767861 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.768475 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.768587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.773042 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.773049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.773054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.773090 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.776166 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.776980 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.776984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.776986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.777097 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.777487 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.777491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.777493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.777737 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.777923 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.778287 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.778507 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":339,"threads_started":1}
I20260927 13:45:59.780665 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.765014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:42623 (local address 127.0.0.1:53636)
0927 13:45:59.765157 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.765174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.765269 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.767841 (+ 2572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.767845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.768179 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.768308 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.768315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.769618 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.769624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.771025 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.771040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.771835 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.771841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.772066 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.772584 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.772600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.774936 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.778236 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.778241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.778245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.778704 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.779050 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.779054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.779058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.779253 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.779736 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.779741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.779914 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.780077 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.780335 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 13:45:59.780746 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.767340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.1.237.253:42623)
0927 13:45:59.767565 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.767569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.767586 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:45:59.767631 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.767634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.767689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.767812 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.768441 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.769457 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.771223 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.771667 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.771801 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.775109 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.775114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.775117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.775193 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.778112 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.778803 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.778806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.778808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.778927 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.779369 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.779375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.779378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.779613 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.779754 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.780161 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.780361 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:45:59.781612 15615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:45:59.798416 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.782595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.1.237.252:44555)
0927 13:45:59.782735 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.782739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.782767 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:45:59.782810 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.782813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.782857 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.782951 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.783669 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.784561 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.786719 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.787389 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.787501 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.792889 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.792896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.792900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.792936 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.795780 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.796353 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.796359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.796363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.796468 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.796857 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.796862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.796866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.797249 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.797414 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.797812 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.798056 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 13:45:59.798678 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.782375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38502)
0927 13:45:59.782519 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.782537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.782628 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.782965 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.782969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.783421 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.783540 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.783548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.784712 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.784720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.786564 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.786582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.788366 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.788375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.788660 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.789360 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.789389 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.792693 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.795915 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.795920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.795924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.796240 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.796607 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.796612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.796615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.796738 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.797400 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.797403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.797529 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.797722 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.798264 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":185}
I20260927 13:45:59.800014 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.780571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.1.237.253:42623)
0927 13:45:59.781000 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.781005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.781020 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:45:59.781344 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.781347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.781432 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.781525 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.784207 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.785155 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.786627 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.787376 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.787501 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.793100 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.793110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.793118 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.793170 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.796867 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.797648 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.797655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.797661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.797816 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.798242 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.798247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.798250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.798596 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.798877 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.799385 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.799543 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":15413,"threads_started":1}
I20260927 13:45:59.804633 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: Bootstrap starting.
I20260927 13:45:59.805025 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:45:59.807978 15615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:59.808995 15615 log.cc:824] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:59.812731 15615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: No bootstrap required, opened a new log
I20260927 13:45:59.823704 15615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.824306 15615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:59.824509 15615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a870c6729e643599673c4bc02ef5b90, State: Initialized, Role: FOLLOWER
I20260927 13:45:59.825381 15615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.830142 15630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:45:59.831972 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.810224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.1.237.254:41653)
0927 13:45:59.810514 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.810524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.813270 (+ 2746us) server_negotiation.cc:407] Connection header received
0927 13:45:59.813499 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.813507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.813613 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.813738 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.815988 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.817041 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.819302 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.819923 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.820018 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.823646 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.823653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.823661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.823705 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.827589 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.828833 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.828841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.828847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.829386 (+ 539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.829892 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.829897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.829900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.830541 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.830766 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.831391 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.831622 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 13:45:59.832458 15630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:45:59.832329 15615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:45:59.835443 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.812018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.1.237.252:44555)
0927 13:45:59.812217 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.812223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.815781 (+ 3558us) server_negotiation.cc:407] Connection header received
0927 13:45:59.816026 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.816031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.816113 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.816184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.819033 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.820560 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.823196 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.824140 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.824260 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.828288 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.828295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.828301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.828341 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.831953 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.832794 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.832801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.832804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.832975 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.833475 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.833482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.833484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.833812 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.834049 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.834634 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.834922 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:45:59.843333 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.815877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.1.237.253:42623)
0927 13:45:59.816051 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.816057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.826633 (+ 10576us) server_negotiation.cc:407] Connection header received
0927 13:45:59.826783 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.826790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.826882 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.826984 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.829104 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.830196 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.832211 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.832902 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.832991 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.837493 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.837499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.837505 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.837552 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.840734 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.841384 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.841389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.841393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.841556 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.841902 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.841907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.841910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.842167 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.842411 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.842726 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.842959 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 13:45:59.849947 15641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:45:59.850121 15641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:45:59.841261 15548 master_runner.cc:186] Error getting master registration for 127.1.237.254:41653: OK, instance_id { permanent_uuid: "55fa6348464042cab82b66168e51302b" instance_seqno: 1790516758759831 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:45:59.852880 15548 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:45:59.893106 15467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 4
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:45:59.908763 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.893872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.1.237.252:44555)
0927 13:45:59.894063 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.894069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.894327 (+ 258us) server_negotiation.cc:407] Connection header received
0927 13:45:59.894379 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.894384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.894458 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.894548 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.895717 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.896877 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.898075 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.898841 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.898968 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.902210 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.902216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.902221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.902264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.905690 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.906525 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.906533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.906537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.906692 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.907048 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.907054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.907057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.907480 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.907633 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.908106 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.908350 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:45:59.909736 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.893752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38516)
0927 13:45:59.894052 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.894073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.894202 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.894729 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.894735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.895350 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.895584 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.895600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.897018 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.897022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.897942 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.897953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.898984 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.898988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.899118 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.899799 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.899817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.902047 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.905831 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.905835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.905839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.906358 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.906809 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.906814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.906818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.906947 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.907634 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.907640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.907837 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.907999 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.908311 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 13:45:59.914184 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: Bootstrap starting.
I20260927 13:45:59.917193 15467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: Neither blocks nor log segments found. Creating new log.
I20260927 13:45:59.918154 15467 log.cc:824] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:45:59.920879 15467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: No bootstrap required, opened a new log
I20260927 13:45:59.928460 15467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.928717 15467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:45:59.928867 15467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fa6348464042cab82b66168e51302b, State: Initialized, Role: FOLLOWER
I20260927 13:45:59.929409 15467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.930265 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:45:59.930657 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:45:59.931464 15467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:45:59.937593 15655 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:45:59.940881 15644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:45:59.941125 15644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.942824 15644 leader_election.cc:288] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 122195df41a24e4d947c15969033716d (127.1.237.253:42623), 6a870c6729e643599673c4bc02ef5b90 (127.1.237.252:44555)
I20260927 13:45:59.941116 15400 master_runner.cc:186] Error getting master registration for 127.1.237.253:42623: OK, instance_id { permanent_uuid: "122195df41a24e4d947c15969033716d" instance_seqno: 1790516759246303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:45:59.947705 15400 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:45:59.948385 15590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55fa6348464042cab82b66168e51302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a870c6729e643599673c4bc02ef5b90" is_pre_election: true
I20260927 13:45:59.948899 15590 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fa6348464042cab82b66168e51302b in term 0.
I20260927 13:45:59.949465 15419 leader_election.cc:302] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fa6348464042cab82b66168e51302b, 6a870c6729e643599673c4bc02ef5b90; no voters:
I20260927 13:45:59.949844 15644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:45:59.949985 15644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:45:59.950083 15644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:59.953234 15644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.954043 15644 leader_election.cc:288] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [CANDIDATE]: Term 1 election: Requested vote from peers 122195df41a24e4d947c15969033716d (127.1.237.253:42623), 6a870c6729e643599673c4bc02ef5b90 (127.1.237.252:44555)
I20260927 13:45:59.955327 15590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55fa6348464042cab82b66168e51302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a870c6729e643599673c4bc02ef5b90"
I20260927 13:45:59.955545 15590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:45:59.958804 15590 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fa6348464042cab82b66168e51302b in term 1.
I20260927 13:45:59.959205 15419 leader_election.cc:302] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fa6348464042cab82b66168e51302b, 6a870c6729e643599673c4bc02ef5b90; no voters:
I20260927 13:45:59.959528 15644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:45:59.960074 15644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [term 1 LEADER]: Becoming Leader. State: Replica: 55fa6348464042cab82b66168e51302b, State: Running, Role: LEADER
I20260927 13:45:59.960564 15644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:45:59.962812 15656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fa6348464042cab82b66168e51302b. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:45:59.963173 15656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: This master's current role is: LEADER
I20260927 13:45:59.964008 15658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 13:45:59.964043 15569 tablet.cc:2369] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:45:59.967267 15541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } attempt: 5
I20260927 13:45:59.969506 15658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 13:45:59 dist-test-slave-7g3l krb5kdc[15387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516759, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:45:59.983942 15590 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [term 1 FOLLOWER]: Refusing update from remote peer 55fa6348464042cab82b66168e51302b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:45:59.984979 15656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:45:59.985224 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.967836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:44555 (local address 127.0.0.1:38532)
0927 13:45:59.968023 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 13:45:59.968043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:45:59.968162 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:45:59.968640 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:45:59.968645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:45:59.969153 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:45:59.969374 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.969386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.970823 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.970828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.972195 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:45:59.972209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.973742 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.973747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:45:59.973994 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.974635 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:45:59.974658 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:45:59.977462 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:45:59.981720 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.981726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.981731 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.982257 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.983291 (+ 1034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:45:59.983296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:45:59.983299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:45:59.983499 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:45:59.984202 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:45:59.984207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:45:59.984376 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 13:45:59.984539 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:45:59.984831 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:45:59.986169 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:45:59.968006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.1.237.252:44555)
0927 13:45:59.968162 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 13:45:59.968167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:45:59.968184 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:45:59.968289 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:45:59.968294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:45:59.968372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:45:59.968456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:45:59.969542 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.970621 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.972390 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:45:59.973038 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:45:59.973189 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:45:59.977635 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:45:59.977643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:45:59.977649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:45:59.977685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:45:59.981596 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.982937 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.982944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.982948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.983107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:45:59.983674 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:45:59.983679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:45:59.983682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:45:59.984075 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:45:59.984247 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 13:45:59.985496 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0927 13:45:59.985783 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:45:59.994971 15541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: Bootstrap starting.
W20260927 13:46:00.000053 15421 tablet.cc:2369] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:46:00.005414 15541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:00.007673 15541 log.cc:824] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:00.012391 15630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fa6348464042cab82b66168e51302b. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.012704 15630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:46:00.013729 15660 mvcc.cc:205] Tried to move back new op lower bound from 7333956648883818496 to 7333956648801570816. Current Snapshot: MvccSnapshot[applied={T|T < 7333956648883818496}]
I20260927 13:46:00.015332 15630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.015538 15630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:46:00.027493 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.027889 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:00.031853 15658 catalog_manager.cc:1347] Generated new cluster ID: 2e446812e3ff437c944d5539a3f43291
I20260927 13:46:00.031961 15658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:00.060305 15541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: No bootstrap required, opened a new log
I20260927 13:46:00.079983 15658 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:46:00.083281 15658 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:00.101508 15541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:46:00.103206 15541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:00.103701 15541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122195df41a24e4d947c15969033716d, State: Initialized, Role: FOLLOWER
I20260927 13:46:00.105928 15541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } }
I20260927 13:46:00.108395 15666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.109124 15666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:46:00.113967 15515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55fa6348464042cab82b66168e51302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122195df41a24e4d947c15969033716d"
I20260927 13:46:00.114372 15514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:00.115231 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55fa6348464042cab82b66168e51302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122195df41a24e4d947c15969033716d" is_pre_election: true
I20260927 13:46:00.115510 15541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:00.121879 15658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b: Generated new TSK 0
I20260927 13:46:00.122995 15658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:46:00.140424 15514 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [term 1 FOLLOWER]: Refusing update from remote peer 55fa6348464042cab82b66168e51302b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:00.143460 15656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [LEADER]: Connected to new peer: Peer: permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:00.211095 15666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fa6348464042cab82b66168e51302b. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.211490 15666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:46:00.211884 15673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.212399 15673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:46:00.250027 15687 catalog_manager.cc:1259] Loaded cluster ID: 2e446812e3ff437c944d5539a3f43291
I20260927 13:46:00.250222 15687 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: loading cluster ID for follower catalog manager: success
I20260927 13:46:00.254154 15687 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: acquiring CA information for follower catalog manager: success
I20260927 13:46:00.257014 15687 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 122195df41a24e4d947c15969033716d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:46:00.564054 15656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55fa6348464042cab82b66168e51302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fa6348464042cab82b66168e51302b" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41653 } } peers { permanent_uuid: "122195df41a24e4d947c15969033716d" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 42623 } } peers { permanent_uuid: "6a870c6729e643599673c4bc02ef5b90" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 44555 } } }
I20260927 13:46:00.564440 15656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa6348464042cab82b66168e51302b [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:00.852164 15641 catalog_manager.cc:1259] Loaded cluster ID: 2e446812e3ff437c944d5539a3f43291
I20260927 13:46:00.852290 15641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: loading cluster ID for follower catalog manager: success
I20260927 13:46:00.854544 15641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: acquiring CA information for follower catalog manager: success
I20260927 13:46:00.856264 15641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a870c6729e643599673c4bc02ef5b90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:46:01.281453 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15400
I20260927 13:46:01.282598 15462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:01.390571 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15400
I20260927 13:46:01.410351 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15474
I20260927 13:46:01.411393 15536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:01.517427 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15474
I20260927 13:46:01.536643 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15548
I20260927 13:46:01.537717 15610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:01.638476 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15548
2026-09-27T13:46:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4576 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:46:01 dist-test-slave-7g3l krb5kdc[15695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:46:01 dist-test-slave-7g3l krb5kdc[15695](info): set up 2 sockets
Sep 27 13:46:01 dist-test-slave-7g3l krb5kdc[15695](info): commencing operation
krb5kdc: starting...
W20260927 13:46:02.699527 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:46:02 dist-test-slave-7g3l krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:46:02.912225 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36943
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:03.242035 15708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:03.242316 15708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:03.242462 15708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:03.249549 15708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:03.249687 15708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:03.249761 15708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:03.249847 15708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:03.249943 15708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:03.259455 15708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:03.260702 15708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:03.261984 15708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:03.273252 15713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:03.273209 15714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:03.273341 15716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:03.274263 15708 server_base.cc:1034] running on GCE node
I20260927 13:46:03.275043 15708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:03.276404 15708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:03.277626 15708 hybrid_clock.cc:648] HybridClock initialized: now 1790516763277588 us; error 44 us; skew 500 ppm
I20260927 13:46:03.277948 15708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:03 dist-test-slave-7g3l krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516763, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:03.280990 15708 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:03.282423 15708 webserver.cc:460] Webserver started at http://127.1.237.254:41791/ using document root <none> and password file <none>
I20260927 13:46:03.283066 15708 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:03.283216 15708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:03.283527 15708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:03.286303 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "4c63cbe0578e456ab475cb55278f2f23"
format_stamp: "Formatted at 2026-09-27 13:46:03 on dist-test-slave-7g3l"
server_key: "b082a655bcd2746fac9c35cf988ccfa5"
server_key_iv: "291624dd52d05803551da2090e8c7b56"
server_key_version: "encryptionkey@0"
I20260927 13:46:03.287052 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "4c63cbe0578e456ab475cb55278f2f23"
format_stamp: "Formatted at 2026-09-27 13:46:03 on dist-test-slave-7g3l"
server_key: "b082a655bcd2746fac9c35cf988ccfa5"
server_key_iv: "291624dd52d05803551da2090e8c7b56"
server_key_version: "encryptionkey@0"
I20260927 13:46:03.291862 15708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:46:03.295058 15723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:03.296295 15708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 13:46:03.296486 15708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "4c63cbe0578e456ab475cb55278f2f23"
format_stamp: "Formatted at 2026-09-27 13:46:03 on dist-test-slave-7g3l"
server_key: "b082a655bcd2746fac9c35cf988ccfa5"
server_key_iv: "291624dd52d05803551da2090e8c7b56"
server_key_version: "encryptionkey@0"
I20260927 13:46:03.296663 15708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:03.317298 15708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:03.320971 15708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:03.321246 15708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:03.338702 15708 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36943
I20260927 13:46:03.338721 15774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36943 every 8 connection(s)
I20260927 13:46:03.340034 15708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:03.341816 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15708
I20260927 13:46:03.342013 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:46:03.342670 1975 external_mini_cluster.cc:1428] Setting key 9aa88c7f96f85e4586b61fe5b2a6e58f
I20260927 13:46:03.344012 15775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:46:03 dist-test-slave-7g3l krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:03.357342 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23: Bootstrap starting.
I20260927 13:46:03.360414 15775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:03.361352 15775 log.cc:824] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:03.363992 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23: No bootstrap required, opened a new log
I20260927 13:46:03.364114 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:03.344353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.1.237.254:36943)
0927 13:46:03.345154 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:03.345164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:03.345203 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:46:03.346755 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:03.346787 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:03.347131 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:03.347503 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:03.349316 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:03.350539 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:03.352338 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:03.352915 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:03.353029 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:03.357539 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:03.357546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:03.357561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:03.357590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:03.360291 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:03.361040 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:03.361046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:03.361052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:03.361200 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:03.361557 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:03.361563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:03.361566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:03.362022 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:03.362340 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:03.362739 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 13:46:03.363112 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":364,"threads_started":1}
I20260927 13:46:03.367518 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:46:03.371479 15775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } }
I20260927 13:46:03.371830 15775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:03.371950 15775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c63cbe0578e456ab475cb55278f2f23, State: Initialized, Role: FOLLOWER
I20260927 13:46:03.372426 15775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } }
I20260927 13:46:03.372639 15775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:03.372772 15775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:03.372931 15775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:03.374850 15775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } }
I20260927 13:46:03.375416 15775 leader_election.cc:302] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c63cbe0578e456ab475cb55278f2f23; no voters:
I20260927 13:46:03.375949 15775 leader_election.cc:288] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:46:03.376052 15780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:03.376780 15780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [term 1 LEADER]: Becoming Leader. State: Replica: 4c63cbe0578e456ab475cb55278f2f23, State: Running, Role: LEADER
I20260927 13:46:03.377281 15780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } }
I20260927 13:46:03.377954 15775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:03.379832 15782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c63cbe0578e456ab475cb55278f2f23. Latest consensus state: current_term: 1 leader_uuid: "4c63cbe0578e456ab475cb55278f2f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } } }
I20260927 13:46:03.379806 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c63cbe0578e456ab475cb55278f2f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c63cbe0578e456ab475cb55278f2f23" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36943 } } }
I20260927 13:46:03.380232 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:03.380285 15782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:03.381680 15785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:03.386373 15785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:03.394515 15785 catalog_manager.cc:1347] Generated new cluster ID: c2b52a25de72430c8ff6da35a0d1f02b
I20260927 13:46:03.394609 15785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:03.409291 15785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:46:03.410502 15785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:03.416450 15785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c63cbe0578e456ab475cb55278f2f23: Generated new TSK 0
I20260927 13:46:03.417140 15785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:46:03.427409 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15708
I20260927 13:46:03.428517 15770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:03.517131 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15708
2026-09-27T13:46:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1874 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:46:03 dist-test-slave-7g3l krb5kdc[15803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:46:03 dist-test-slave-7g3l krb5kdc[15803](info): set up 2 sockets
Sep 27 13:46:03 dist-test-slave-7g3l krb5kdc[15803](info): commencing operation
krb5kdc: starting...
W20260927 13:46:04.587625 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:46:04 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:46:04.806285 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:05.120401 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:05.120697 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:05.120850 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:05.127373 15816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:05.127501 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:05.127600 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:05.127687 15816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:05.127750 15816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:05.136704 15816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:43241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:05.137899 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:05.139086 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:05.150700 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:05.150662 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:05.151374 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:05.151731 15816 server_base.cc:1034] running on GCE node
I20260927 13:46:05.152442 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:05.153676 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:05.154867 15816 hybrid_clock.cc:648] HybridClock initialized: now 1790516765154821 us; error 34 us; skew 500 ppm
I20260927 13:46:05.155139 15816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:05 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:05.164537 15816 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:05.165931 15816 webserver.cc:460] Webserver started at http://127.1.237.254:46217/ using document root <none> and password file <none>
I20260927 13:46:05.166601 15816 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:05.166731 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:05.167032 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:05.170020 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "2b013e8a7f7a4bf2b6adaadce559b71f"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "b8696c7b45e6638a9bcb4bc2cb755826"
server_key_iv: "7a3f635d634572a502300939b939c316"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.170743 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "2b013e8a7f7a4bf2b6adaadce559b71f"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "b8696c7b45e6638a9bcb4bc2cb755826"
server_key_iv: "7a3f635d634572a502300939b939c316"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.175768 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 13:46:05.178933 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:05.180203 15816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:05.180392 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "2b013e8a7f7a4bf2b6adaadce559b71f"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "b8696c7b45e6638a9bcb4bc2cb755826"
server_key_iv: "7a3f635d634572a502300939b939c316"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.180574 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:05.202167 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:05.205816 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:05.206080 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:05.223282 15816 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:43241
I20260927 13:46:05.223302 15882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:43241 every 8 connection(s)
I20260927 13:46:05.224527 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:05.226642 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15816
I20260927 13:46:05.226861 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:46:05.227598 1975 external_mini_cluster.cc:1428] Setting key 924346516fcc49a0b1e161e8e15f720c
I20260927 13:46:05.228154 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:05.242098 15883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap starting.
Sep 27 13:46:05 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:05.245786 15883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:05.247112 15883 log.cc:824] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:05.250389 15883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: No bootstrap required, opened a new log
I20260927 13:46:05.251655 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:05.229408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.1.237.254:43241)
0927 13:46:05.230122 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:05.230132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:05.230170 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:46:05.232526 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:05.232546 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:05.232896 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:05.233213 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:05.234960 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.236272 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.238053 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.238735 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.238869 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:05.243543 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:05.243559 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:05.243579 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:05.243625 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:05.247452 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:05.248315 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:05.248320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:05.248325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:05.248527 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:05.248886 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:05.248890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:05.248893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:05.249528 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:05.249833 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:05.250098 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 13:46:05.250493 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260927 13:46:05.258843 15883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:05.259214 15883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:05.259328 15883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Initialized, Role: FOLLOWER
I20260927 13:46:05.259886 15883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:05.260113 15883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:05.260249 15883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:05.260421 15883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:05.262758 15883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:05.263288 15883 leader_election.cc:302] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b013e8a7f7a4bf2b6adaadce559b71f; no voters:
I20260927 13:46:05.263835 15883 leader_election.cc:288] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:46:05.263975 15888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:05.265039 15888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 LEADER]: Becoming Leader. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Running, Role: LEADER
I20260927 13:46:05.265507 15888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:05.265702 15883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:05.268435 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:05.268894 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:05.268684 15890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b013e8a7f7a4bf2b6adaadce559b71f. Latest consensus state: current_term: 1 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:05.269369 15890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:05.269932 15897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:05.278909 15897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:05.287103 15897 catalog_manager.cc:1347] Generated new cluster ID: ce13067eba6f4f31b93be6b3014b466f
I20260927 13:46:05.287215 15897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:05.322521 15897 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:46:05.323830 15897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:05.334988 15897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Generated new TSK 0
I20260927 13:46:05.335835 15897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:46:05.395283 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:05.719978 15909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:05.720240 15909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:05.720408 15909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:05.727584 15909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:05.727685 15909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:05.727878 15909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:46:05.737367 15909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:05.738605 15909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:05.739950 15909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:05.751824 15914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:05.752211 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:05.752394 15915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:05.753434 15909 server_base.cc:1034] running on GCE node
I20260927 13:46:05.753993 15909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:05.754848 15909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:05.756132 15909 hybrid_clock.cc:648] HybridClock initialized: now 1790516765756070 us; error 46 us; skew 500 ppm
I20260927 13:46:05.756489 15909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:05 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:05.759531 15909 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:46:05.760967 15909 webserver.cc:460] Webserver started at http://127.1.237.193:40347/ using document root <none> and password file <none>
I20260927 13:46:05.761644 15909 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:05.761787 15909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:05.762068 15909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:05.764892 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "3317559ec8d04297b624d98034b1d5c6"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "594ac244bbd747c2ac873b8b75ae5d05"
server_key_iv: "b1358b470bc9ca32879ea9aec0e60eeb"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.765681 15909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "3317559ec8d04297b624d98034b1d5c6"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "594ac244bbd747c2ac873b8b75ae5d05"
server_key_iv: "b1358b470bc9ca32879ea9aec0e60eeb"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.770819 15909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:46:05.774206 15924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:05.775635 15909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:05.775801 15909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "3317559ec8d04297b624d98034b1d5c6"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "594ac244bbd747c2ac873b8b75ae5d05"
server_key_iv: "b1358b470bc9ca32879ea9aec0e60eeb"
server_key_version: "encryptionkey@0"
I20260927 13:46:05.775990 15909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:05.798611 15909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:05.802439 15909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:05.802728 15909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:05.803721 15909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:05.805034 15909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:05.805140 15909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:05.805284 15909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:05.805354 15909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:05.837991 15909 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:33683
I20260927 13:46:05.838030 16036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:33683 every 8 connection(s)
I20260927 13:46:05.839298 15909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:46:05.846926 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 15909
I20260927 13:46:05.847121 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:46:05.847754 1975 external_mini_cluster.cc:1428] Setting key 7360e86e91fd6de886ad11a15f84772f
Sep 27 13:46:05 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:05.858538 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:05.841685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:46903 (local address 127.1.237.254:43241)
0927 13:46:05.842137 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:05.842142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:05.842939 (+ 797us) server_negotiation.cc:407] Connection header received
0927 13:46:05.844803 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:05.844809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:05.844896 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:05.845008 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:05.846495 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.847307 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.848964 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.849436 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.849546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:05.852995 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:05.853000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:05.853003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:05.853031 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:05.855517 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:05.856214 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:05.856220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:05.856223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:05.856347 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:05.856812 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:05.856818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:05.856824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:05.857209 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:05.857383 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:05.858083 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 13:46:05.858261 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":250,"threads_started":1}
I20260927 13:46:05.860026 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:05.842112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:46903)
0927 13:46:05.842789 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:05.842831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:05.844531 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:05.845234 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:05.845249 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:05.845716 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:05.846333 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:05.846351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.847460 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.847465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:05.848806 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:05.848821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:05.849661 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:05.849671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:05.849887 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:05.850541 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:05.850564 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:05.852791 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:05.855647 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:05.855655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:05.855670 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:05.856096 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:05.856522 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:05.856526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:05.856528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:05.856704 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:05.857338 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:05.857343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:05.857687 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 13:46:05.857984 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 13:46:05.858563 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":257,"threads_started":1}
I20260927 13:46:05.861545 16037 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:05.861886 16037 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:05.862670 16037 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:05.864801 15848 ts_manager.cc:194] Registered new tserver with Master: 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683)
I20260927 13:46:05.866314 15848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:46903
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:46:05.900192 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:06.208250 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:06.208509 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:06.208658 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:06.215445 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:06.215540 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:06.215726 16043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:46:06.224886 16043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:06.225996 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:06.227295 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:06.240501 16048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:06.240536 16051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:06.240454 16049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:06.241557 16043 server_base.cc:1034] running on GCE node
I20260927 13:46:06.242136 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:06.242969 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:06.244177 16043 hybrid_clock.cc:648] HybridClock initialized: now 1790516766244124 us; error 41 us; skew 500 ppm
I20260927 13:46:06.244474 16043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:06 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516766, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:06.247449 16043 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:46:06.248760 16043 webserver.cc:460] Webserver started at http://127.1.237.194:42889/ using document root <none> and password file <none>
I20260927 13:46:06.249393 16043 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:06.249524 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:06.249809 16043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:06.252571 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "7e8254dccc0d4268a38028e118c73aa4"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "4a7077ee1d5dc10c7bb6525d2a660de3"
server_key_iv: "4347955d1a0bc3988430b18a20e115bf"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.253317 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "7e8254dccc0d4268a38028e118c73aa4"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "4a7077ee1d5dc10c7bb6525d2a660de3"
server_key_iv: "4347955d1a0bc3988430b18a20e115bf"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.258175 16043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:46:06.261381 16058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.262594 16043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:46:06.262763 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "7e8254dccc0d4268a38028e118c73aa4"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "4a7077ee1d5dc10c7bb6525d2a660de3"
server_key_iv: "4347955d1a0bc3988430b18a20e115bf"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.262956 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:06.284188 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:06.288069 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:06.288384 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:06.289307 16043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:06.290547 16043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:06.290620 16043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.290783 16043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:06.290833 16043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.322101 16043 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:34819
I20260927 13:46:06.322144 16170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:34819 every 8 connection(s)
I20260927 13:46:06.323354 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:46:06.328958 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16043
I20260927 13:46:06.329138 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:46:06.329572 1975 external_mini_cluster.cc:1428] Setting key 605a5dc43777eb26519c7877004c27c9
Sep 27 13:46:06 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516766, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:06.344496 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.325734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:53099 (local address 127.1.237.254:43241)
0927 13:46:06.325914 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.325919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.326978 (+ 1059us) server_negotiation.cc:407] Connection header received
0927 13:46:06.328688 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.328694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.328779 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.328882 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:06.330467 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.331535 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.333015 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.333416 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.333534 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.337765 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:06.337773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:06.337780 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:06.337819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:06.341357 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.342048 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.342056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.342062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.342226 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.342592 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.342599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.342604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.342979 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:06.343129 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.343812 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.344076 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:46:06.345085 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.326167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:53099)
0927 13:46:06.326827 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:06.326861 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:06.328410 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:06.329133 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:06.329144 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:06.329638 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:06.330321 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.330341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.331682 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.331687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.332879 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.332890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.333627 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.333634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.333845 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.334536 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:06.334567 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:06.337557 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:06.341497 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:06.341504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:06.341515 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:06.341936 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:06.342330 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:06.342334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:06.342336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:06.342496 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:06.343109 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:06.343115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:06.343460 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 13:46:06.343671 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 13:46:06.344031 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":256,"threads_started":1}
I20260927 13:46:06.346565 16171 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:06.346930 16171 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:06.347652 16171 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:06.349080 15848 ts_manager.cc:194] Registered new tserver with Master: 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
I20260927 13:46:06.350317 15848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:53099
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:46:06.379529 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:06.685813 16176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:06.686075 16176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:06.686256 16176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:06.693269 16176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:06.693368 16176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:06.693553 16176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:46:06.702852 16176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:06.704125 16176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:06.705441 16176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:06.718021 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:06.718128 16182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:06.718107 16184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:06.718873 16176 server_base.cc:1034] running on GCE node
I20260927 13:46:06.719554 16176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:06.720441 16176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:06.721640 16176 hybrid_clock.cc:648] HybridClock initialized: now 1790516766721592 us; error 40 us; skew 500 ppm
I20260927 13:46:06.721948 16176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:06 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516766, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:06.724994 16176 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:46:06.726444 16176 webserver.cc:460] Webserver started at http://127.1.237.195:43667/ using document root <none> and password file <none>
I20260927 13:46:06.727093 16176 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:06.727277 16176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:06.727591 16176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:06.730301 16176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "da668c284b3a4420eb7cf9350378e586"
server_key_iv: "966c5a8ce5c2bff2cf1bfd8b903e1aa5"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.731026 16176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "da668c284b3a4420eb7cf9350378e586"
server_key_iv: "966c5a8ce5c2bff2cf1bfd8b903e1aa5"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.735772 16176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:46:06.738879 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.740164 16176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:06.740311 16176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "da668c284b3a4420eb7cf9350378e586"
server_key_iv: "966c5a8ce5c2bff2cf1bfd8b903e1aa5"
server_key_version: "encryptionkey@0"
I20260927 13:46:06.740497 16176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:06.782909 16176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:06.786886 16176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:06.787135 16176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:06.788067 16176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:06.789320 16176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:06.789400 16176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.789556 16176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:06.789634 16176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:06.821667 16176 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:41513
I20260927 13:46:06.821710 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:41513 every 8 connection(s)
I20260927 13:46:06.822842 16176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:46:06.828166 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16176
I20260927 13:46:06.828353 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:46:06.828872 1975 external_mini_cluster.cc:1428] Setting key f04ca60261106e0ac156d31f2952cfac
Sep 27 13:46:06 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516766, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:06.843477 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.825277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:40809 (local address 127.1.237.254:43241)
0927 13:46:06.825469 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.825473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.826629 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 13:46:06.828713 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.828717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.828780 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.828871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:06.830592 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.831546 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.833054 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.833431 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.833550 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.837670 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:06.837676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:06.837680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:06.837717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:06.840282 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.841115 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.841120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.841123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.841264 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.841711 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.841714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.841716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.841973 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:06.842132 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.842894 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.843107 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:46:06.844587 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.825804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:40809)
0927 13:46:06.826469 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:06.826504 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:06.828440 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:06.829122 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:06.829132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:06.829675 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:06.830274 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.830315 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.831703 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.831712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.832914 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.832925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.833627 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.833636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.833842 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.834405 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:06.834426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:06.837362 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:06.840412 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:06.840420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:06.840434 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:06.840974 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:06.841371 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:06.841375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:06.841378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:06.841613 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:06.842110 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:06.842117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:06.842518 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 13:46:06.842805 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 13:46:06.843342 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":250,"threads_started":1}
I20260927 13:46:06.845880 16304 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:06.846166 16304 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:06.846819 16304 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:06.848129 15848 ts_manager.cc:194] Registered new tserver with Master: 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513)
I20260927 13:46:06.848850 15848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:40809
I20260927 13:46:06.855749 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:46:06.869911 16037 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
I20260927 13:46:06.871838 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.859842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.1.237.254:43241)
0927 13:46:06.860047 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.860055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.860351 (+ 296us) server_negotiation.cc:407] Connection header received
0927 13:46:06.860472 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.860476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.860531 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.860605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:06.861807 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.862478 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.863732 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.864138 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.864229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.866537 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:06.866541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:06.866544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:06.866572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:06.869059 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.869772 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.869777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.869780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.869930 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:06.870389 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:06.870394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:06.870399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:06.870749 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:06.871006 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.871428 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.871618 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:46:06.875419 15848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:46:06.882545 15848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:46:06.904321 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.896986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.1.237.193:33683)
0927 13:46:06.897763 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.897772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.897797 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:46:06.897885 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.897892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.898107 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.898917 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:06.899147 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.900352 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.902351 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903259 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.903396 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.903601 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.903831 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.903978 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":480,"threads_started":1}
I20260927 13:46:06.904408 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.896561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50464)
0927 13:46:06.897097 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:06.897114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:06.897219 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:06.898283 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:06.898287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:06.898312 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:06.898738 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.898757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.900493 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.900502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.902186 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.902199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903419 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.903424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.903589 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.903639 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 13:46:06.903781 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 13:46:06.903956 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":22,"thread_start_us":517,"threads_started":1}
I20260927 13:46:06.904539 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.894329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:34819 (local address 127.0.0.1:53234)
0927 13:46:06.895131 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:06.895193 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:06.895373 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:06.897381 (+ 2008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:06.897388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:06.897437 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:06.898085 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.898098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.899967 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.899972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.902052 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.902064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903666 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.903672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.903838 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.903884 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 13:46:06.903943 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 13:46:06.904082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":151,"spinlock_wait_cycles":8704,"thread_start_us":1806,"threads_started":1}
I20260927 13:46:06.904886 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.894347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:41513 (local address 127.0.0.1:36618)
0927 13:46:06.896320 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:06.896333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:06.896463 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:06.897381 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:06.897388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:06.897437 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:06.898027 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.898043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.899642 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.899649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.901918 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:06.901934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903902 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.903907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:06.904081 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.904116 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:46:06.904210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:46:06.904332 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1887,"mutex_wait_us":51,"spinlock_wait_cycles":3328,"thread_start_us":963,"threads_started":1}
I20260927 13:46:06.905135 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.895032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.1.237.195:41513)
0927 13:46:06.895989 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.895998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.896426 (+ 428us) server_negotiation.cc:407] Connection header received
0927 13:46:06.896672 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.896680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.896916 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.897086 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:06.898239 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.899523 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.902113 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903428 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.903573 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.903776 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.904637 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.904780 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":824,"threads_started":1}
I20260927 13:46:06.905652 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:06.894400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.1.237.194:34819)
0927 13:46:06.896844 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:06.896851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:06.896874 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:46:06.896958 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:06.896963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:06.897159 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:06.897721 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:06.898307 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.899844 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.902210 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:06.903460 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:06.904942 (+ 1482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:06.905142 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:06.905203 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 13:46:06.905374 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2309,"thread_start_us":2184,"threads_started":1}
I20260927 13:46:06.908305 16239 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7636eb00442f38a24e7443b85717a (DEFAULT_TABLE table=test-table [id=aad4f724b1634957b2bb4489591aa3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:06.908572 15972 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7636eb00442f38a24e7443b85717a (DEFAULT_TABLE table=test-table [id=aad4f724b1634957b2bb4489591aa3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:06.908774 16106 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd7636eb00442f38a24e7443b85717a (DEFAULT_TABLE table=test-table [id=aad4f724b1634957b2bb4489591aa3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:06.909569 16239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7636eb00442f38a24e7443b85717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:06.909878 15972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7636eb00442f38a24e7443b85717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:06.909898 16106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd7636eb00442f38a24e7443b85717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:06.920527 16322 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Bootstrap starting.
I20260927 13:46:06.923652 16323 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Bootstrap starting.
I20260927 13:46:06.924919 16322 tablet_bootstrap.cc:654] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:06.925014 16324 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Bootstrap starting.
I20260927 13:46:06.926321 16322 log.cc:824] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:06.927215 16323 tablet_bootstrap.cc:654] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:06.927984 16324 tablet_bootstrap.cc:654] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:06.928370 16323 log.cc:824] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:06.928962 16324 log.cc:824] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:06.929442 16322 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: No bootstrap required, opened a new log
I20260927 13:46:06.929714 16322 ts_tablet_manager.cc:1397] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:46:06.931303 16323 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: No bootstrap required, opened a new log
I20260927 13:46:06.931569 16323 ts_tablet_manager.cc:1397] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 13:46:06.931870 16324 tablet_bootstrap.cc:492] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: No bootstrap required, opened a new log
I20260927 13:46:06.932101 16324 ts_tablet_manager.cc:1397] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 13:46:06.937129 16322 raft_consensus.cc:348] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.937455 16322 raft_consensus.cc:374] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:06.937573 16322 raft_consensus.cc:729] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00da0cbc1d8d4bd5aa7a85aeb01c2228, State: Initialized, Role: FOLLOWER
I20260927 13:46:06.938129 16322 consensus_queue.cc:260] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.939201 16304 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
I20260927 13:46:06.939803 16322 ts_tablet_manager.cc:1428] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Time spent starting tablet: real 0.010s user 0.000s sys 0.011s
I20260927 13:46:06.941820 16323 raft_consensus.cc:348] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.942234 16323 raft_consensus.cc:374] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:06.942334 16323 raft_consensus.cc:729] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8254dccc0d4268a38028e118c73aa4, State: Initialized, Role: FOLLOWER
I20260927 13:46:06.942298 16324 raft_consensus.cc:348] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.942695 16324 raft_consensus.cc:374] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:06.942788 16324 raft_consensus.cc:729] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3317559ec8d04297b624d98034b1d5c6, State: Initialized, Role: FOLLOWER
I20260927 13:46:06.942902 16323 consensus_queue.cc:260] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.943424 16324 consensus_queue.cc:260] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:06.944082 16171 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
I20260927 13:46:06.944945 16323 ts_tablet_manager.cc:1428] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 13:46:06.946394 16324 ts_tablet_manager.cc:1428] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
W20260927 13:46:07.074400 16305 tablet.cc:2369] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:46:07.075259 16172 tablet.cc:2369] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:46:07.091411 16038 tablet.cc:2369] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:46:07.111304 16329 raft_consensus.cc:482] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:07.111543 16329 raft_consensus.cc:504] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:07.113021 16329 leader_election.cc:288] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683)
I20260927 13:46:07.121263 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.113529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:40997 (local address 127.1.237.195:41513)
0927 13:46:07.113732 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.113738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.114483 (+ 745us) server_negotiation.cc:407] Connection header received
0927 13:46:07.114602 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.114606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.114663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.114734 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:07.115250 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.116317 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.119002 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.120189 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.120346 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.120415 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.120920 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.121035 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:46:07.121265 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.113516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:51369 (local address 127.1.237.193:33683)
0927 13:46:07.113732 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.113738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.114437 (+ 699us) server_negotiation.cc:407] Connection header received
0927 13:46:07.114905 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.114909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.114970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.115032 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:07.115621 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.116583 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.118997 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.120199 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.120359 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.120420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.120920 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.121035 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:46:07.121479 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.113472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.1.237.194:51369)
0927 13:46:07.114328 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:07.114347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:07.114489 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:07.115076 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:07.115079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:07.115107 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:07.115482 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:07.115490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.116694 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.116699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:07.118844 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:07.118859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.120428 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.120432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:07.120689 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.120789 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 13:46:07.120872 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 13:46:07.121053 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":143,"thread_start_us":479,"threads_started":1}
I20260927 13:46:07.121747 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.113472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:41513 (local address 127.1.237.194:40997)
0927 13:46:07.114328 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:07.114347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:07.114489 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:07.114758 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:07.114761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:07.114779 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:07.115071 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:07.115080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.116463 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.116467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:07.118844 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:07.118859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.120418 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.120425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:07.120692 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.120788 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 13:46:07.120872 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:46:07.121053 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":232,"spinlock_wait_cycles":56064,"thread_start_us":466,"threads_started":1}
I20260927 13:46:07.122406 15992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7636eb00442f38a24e7443b85717a" candidate_uuid: "7e8254dccc0d4268a38028e118c73aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3317559ec8d04297b624d98034b1d5c6" is_pre_election: true
I20260927 13:46:07.122594 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7636eb00442f38a24e7443b85717a" candidate_uuid: "7e8254dccc0d4268a38028e118c73aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" is_pre_election: true
I20260927 13:46:07.122838 15992 raft_consensus.cc:2454] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8254dccc0d4268a38028e118c73aa4 in term 0.
I20260927 13:46:07.123044 16259 raft_consensus.cc:2454] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8254dccc0d4268a38028e118c73aa4 in term 0.
I20260927 13:46:07.123492 16061 leader_election.cc:302] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6, 7e8254dccc0d4268a38028e118c73aa4; no voters:
I20260927 13:46:07.123837 16329 raft_consensus.cc:2781] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:46:07.123971 16329 raft_consensus.cc:482] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:46:07.124089 16329 raft_consensus.cc:3037] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:07.126137 16329 raft_consensus.cc:504] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:07.126710 16329 leader_election.cc:288] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [CANDIDATE]: Term 1 election: Requested vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683)
I20260927 13:46:07.127135 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7636eb00442f38a24e7443b85717a" candidate_uuid: "7e8254dccc0d4268a38028e118c73aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
I20260927 13:46:07.127180 15992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd7636eb00442f38a24e7443b85717a" candidate_uuid: "7e8254dccc0d4268a38028e118c73aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3317559ec8d04297b624d98034b1d5c6"
I20260927 13:46:07.127355 15992 raft_consensus.cc:3037] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:07.127357 16259 raft_consensus.cc:3037] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:07.130378 15992 raft_consensus.cc:2454] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8254dccc0d4268a38028e118c73aa4 in term 1.
I20260927 13:46:07.130426 16259 raft_consensus.cc:2454] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8254dccc0d4268a38028e118c73aa4 in term 1.
I20260927 13:46:07.130781 16060 leader_election.cc:302] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 3317559ec8d04297b624d98034b1d5c6, 7e8254dccc0d4268a38028e118c73aa4; no voters:
I20260927 13:46:07.131067 16329 raft_consensus.cc:2781] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:07.131524 16329 raft_consensus.cc:686] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8254dccc0d4268a38028e118c73aa4, State: Running, Role: LEADER
I20260927 13:46:07.131937 16329 consensus_queue.cc:237] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } }
I20260927 13:46:07.136318 15847 catalog_manager.cc:5495] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "7e8254dccc0d4268a38028e118c73aa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } health_report { overall_health: UNKNOWN } } }
I20260927 13:46:07.195425 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.190758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.1.237.194:34819)
0927 13:46:07.190930 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.190934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.190950 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:46:07.191117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.191121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.191213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.191335 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:07.191631 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.192299 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.193490 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.193854 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.193942 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.194435 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:07.194904 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:46:07.195017 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.195086 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.195145 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:46:07.203007 16259 raft_consensus.cc:1261] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Refusing update from remote peer 7e8254dccc0d4268a38028e118c73aa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:07.203006 15992 raft_consensus.cc:1261] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Refusing update from remote peer 7e8254dccc0d4268a38028e118c73aa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:07.203876 16333 consensus_queue.cc:1035] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:07.204157 16329 consensus_queue.cc:1035] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:07.215917 16338 mvcc.cc:205] Tried to move back new op lower bound from 7333956678453272576 to 7333956678175731712. Current Snapshot: MvccSnapshot[applied={T|T < 7333956678453272576}]
I20260927 13:46:07.232612 15847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:07.232864 15847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:07.236737 15847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Sending DeleteTablet for 3 replicas of tablet 8cd7636eb00442f38a24e7443b85717a
I20260927 13:46:07.237881 15972 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd7636eb00442f38a24e7443b85717a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:50464
I20260927 13:46:07.237928 16239 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd7636eb00442f38a24e7443b85717a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:36618
I20260927 13:46:07.238554 16106 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd7636eb00442f38a24e7443b85717a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:07 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53234
I20260927 13:46:07.238956 16347 tablet_replica.cc:330] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: stopping tablet replica
I20260927 13:46:07.239464 16346 tablet_replica.cc:330] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: stopping tablet replica
I20260927 13:46:07.239401 16347 raft_consensus.cc:2229] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:07.239580 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15909
I20260927 13:46:07.239866 16346 raft_consensus.cc:2229] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:07.240121 16347 raft_consensus.cc:2258] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:07.240293 16346 raft_consensus.cc:2258] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:07.242111 16347 ts_tablet_manager.cc:1905] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:07.243431 16346 ts_tablet_manager.cc:1905] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:07.249164 16347 ts_tablet_manager.cc:1918] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:07.249377 16347 log.cc:1196] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/8cd7636eb00442f38a24e7443b85717a
I20260927 13:46:07.249967 16347 ts_tablet_manager.cc:1939] T 8cd7636eb00442f38a24e7443b85717a P 3317559ec8d04297b624d98034b1d5c6: Deleting consensus metadata
I20260927 13:46:07.250751 16348 tablet_replica.cc:330] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: stopping tablet replica
I20260927 13:46:07.251142 16348 raft_consensus.cc:2229] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:46:07.251773 16348 raft_consensus.cc:2258] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:07.251917 15834 catalog_manager.cc:4841] TS 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683): tablet 8cd7636eb00442f38a24e7443b85717a (table test-table [id=aad4f724b1634957b2bb4489591aa3d0]) successfully deleted
I20260927 13:46:07.251983 16346 ts_tablet_manager.cc:1918] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:07.252118 16346 log.cc:1196] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/8cd7636eb00442f38a24e7443b85717a
I20260927 13:46:07.252624 16346 ts_tablet_manager.cc:1939] T 8cd7636eb00442f38a24e7443b85717a P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting consensus metadata
I20260927 13:46:07.253840 16348 ts_tablet_manager.cc:1905] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:07.254197 15833 catalog_manager.cc:4841] TS 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): tablet 8cd7636eb00442f38a24e7443b85717a (table test-table [id=aad4f724b1634957b2bb4489591aa3d0]) successfully deleted
I20260927 13:46:07.260952 16348 ts_tablet_manager.cc:1918] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:07.261119 16348 log.cc:1196] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/8cd7636eb00442f38a24e7443b85717a
I20260927 13:46:07.261677 16348 ts_tablet_manager.cc:1939] T 8cd7636eb00442f38a24e7443b85717a P 7e8254dccc0d4268a38028e118c73aa4: Deleting consensus metadata
Sep 27 13:46:07 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:46:07.262954 15834 catalog_manager.cc:4841] TS 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): tablet 8cd7636eb00442f38a24e7443b85717a (table test-table [id=aad4f724b1634957b2bb4489591aa3d0]) successfully deleted
I20260927 13:46:07.269174 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.240689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.1.237.193:33683)
0927 13:46:07.240849 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.240856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.247703 (+ 6847us) server_negotiation.cc:407] Connection header received
0927 13:46:07.248546 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.248551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.248630 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.248706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:07.254296 (+ 5590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.255464 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.257561 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.258265 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.258430 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.263656 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:07.263666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:07.263678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:07.263707 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:07.266438 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.267314 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.267322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.267326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.267473 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.267847 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.267850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.267852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.268179 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:07.268384 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.268757 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.268903 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 13:46:07.269902 16032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:07.367316 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15909
I20260927 13:46:07.388294 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16043
Sep 27 13:46:07 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:46:07.403230 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.389071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.1.237.194:34819)
0927 13:46:07.389244 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.389249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.389318 (+ 69us) server_negotiation.cc:407] Connection header received
0927 13:46:07.389503 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.389508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.389576 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.389674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:07.390869 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.391740 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.393003 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.393407 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.393538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.397708 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:07.397719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:07.397735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:07.397772 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:07.400619 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.401257 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.401263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.401265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.401406 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.401781 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.401789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.401792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.402120 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:07.402421 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.402636 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.402826 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:46:07.403857 16166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:07.516705 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16043
I20260927 13:46:07.534474 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16176
Sep 27 13:46:07 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:46:07.548988 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:07.535181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.1.237.195:41513)
0927 13:46:07.535373 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:07.535377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:07.535396 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:46:07.535552 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:07.535556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:07.535619 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:07.535739 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:07.537016 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.537706 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.539026 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:07.539582 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:07.539673 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:07.544110 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:07.544122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:07.544138 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:07.544167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:07.546571 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.547211 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.547218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.547220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.547358 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:07.547701 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:07.547704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:07.547706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:07.548060 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:07.548265 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:07.548489 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 13:46:07.548643 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:46:07.549695 16299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:07.636916 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16176
I20260927 13:46:07.653206 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 15816
I20260927 13:46:07.654214 15878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:07.763011 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 15816
I20260927 13:46:07.783372 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--webserver_interface=127.1.237.254
--webserver_port=46217
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:08.107338 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:08.107616 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:08.107703 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:08.114553 16354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:08.114674 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:08.114717 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:08.114763 16354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:08.114796 16354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:08.124178 16354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:43241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=46217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:08.125367 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:08.126547 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:08.148527 16360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:08.148617 16363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:08.148590 16361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:08.149519 16354 server_base.cc:1034] running on GCE node
I20260927 13:46:08.150520 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:08.152211 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:08.153452 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790516768153383 us; error 59 us; skew 500 ppm
I20260927 13:46:08.153853 16354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:08 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:08.157299 16354 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:08.159049 16354 webserver.cc:460] Webserver started at http://127.1.237.254:46217/ using document root <none> and password file <none>
I20260927 13:46:08.159816 16354 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:08.159967 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:08.165729 16354 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:46:08.168813 16370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:08.170306 16354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 13:46:08.170523 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "2b013e8a7f7a4bf2b6adaadce559b71f"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "b8696c7b45e6638a9bcb4bc2cb755826"
server_key_iv: "7a3f635d634572a502300939b939c316"
server_key_version: "encryptionkey@0"
I20260927 13:46:08.171697 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:08.191982 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:08.196105 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:08.196403 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:08.215319 16354 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:43241
I20260927 13:46:08.215365 16421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:43241 every 8 connection(s)
I20260927 13:46:08.216830 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:08.223594 16422 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:46:08.224789 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16354
I20260927 13:46:08.225554 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:33683
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=40347
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:46:08.226783 16422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap starting.
I20260927 13:46:08.245641 16422 log.cc:824] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:08.257225 16422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:46:08.257763 16422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap complete.
I20260927 13:46:08.266571 16422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:08.267419 16422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Initialized, Role: FOLLOWER
I20260927 13:46:08.267951 16422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:08.268180 16422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:08.268296 16422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:08.268469 16422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:46:08.272181 16422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:08.272665 16422 leader_election.cc:302] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b013e8a7f7a4bf2b6adaadce559b71f; no voters:
I20260927 13:46:08.273286 16422 leader_election.cc:288] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:46:08.273423 16426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:46:08.274871 16426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 LEADER]: Becoming Leader. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Running, Role: LEADER
I20260927 13:46:08.275501 16426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:08.275671 16422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:08.277837 16428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b013e8a7f7a4bf2b6adaadce559b71f. Latest consensus state: current_term: 2 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:08.278229 16428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:08.278728 16427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:08.279012 16427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:08.289032 16432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:08.296952 16432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:08.298349 16432 catalog_manager.cc:1259] Loaded cluster ID: ce13067eba6f4f31b93be6b3014b466f
I20260927 13:46:08.298507 16432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:08.301530 16432 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 13:46:08.319314 16432 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 13:46:08.329704 16443 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:46:08.334131 16432 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:08.335824 16432 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Loaded TSK: 0
I20260927 13:46:08.337224 16432 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:46:08.563799 16424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:08.564047 16424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:08.564199 16424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:08.571376 16424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:08.571473 16424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:08.571648 16424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:46:08.581212 16424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:33683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=40347
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:08.582453 16424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:08.583798 16424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:08.598598 16450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:08.599284 16449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:08.599448 16452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:08.600468 16424 server_base.cc:1034] running on GCE node
I20260927 13:46:08.601063 16424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:08.602068 16424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:08.603248 16424 hybrid_clock.cc:648] HybridClock initialized: now 1790516768603191 us; error 49 us; skew 500 ppm
I20260927 13:46:08.603544 16424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:08 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:08.606585 16424 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:46:08.608112 16424 webserver.cc:460] Webserver started at http://127.1.237.193:40347/ using document root <none> and password file <none>
I20260927 13:46:08.608767 16424 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:08.608901 16424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:08.614138 16424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:46:08.616945 16459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:08.618328 16424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:46:08.618495 16424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "3317559ec8d04297b624d98034b1d5c6"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "594ac244bbd747c2ac873b8b75ae5d05"
server_key_iv: "b1358b470bc9ca32879ea9aec0e60eeb"
server_key_version: "encryptionkey@0"
I20260927 13:46:08.619488 16424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:08.636708 16424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:08.640910 16424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:08.641177 16424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:08.642207 16424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:08.643576 16424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:08.643694 16424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:08.643811 16424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:08.643904 16424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:08.676399 16424 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:33683
I20260927 13:46:08.676474 16571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:33683 every 8 connection(s)
I20260927 13:46:08.677738 16424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:46:08.686839 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16424
I20260927 13:46:08.687462 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:34819
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=42889
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:46:08 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:08.700942 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:08.680697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:38335)
0927 13:46:08.681410 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:08.681444 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:08.682992 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:08.684319 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:08.684327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:08.684806 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:08.685310 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:08.685327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:08.686872 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:08.686877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:08.688194 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:08.688207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:08.689203 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:08.689222 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:08.689486 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:08.690166 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:08.690202 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:08.692886 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:08.697114 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:08.697120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:08.697130 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:08.697528 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:08.697992 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:08.697995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:08.697997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:08.698177 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:08.699069 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:08.699074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:08.699435 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 13:46:08.699711 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 13:46:08.700046 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":300,"threads_started":1}
I20260927 13:46:08.701606 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:08.680369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:38335 (local address 127.1.237.254:43241)
0927 13:46:08.680871 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:08.680895 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:08.681567 (+ 672us) server_negotiation.cc:407] Connection header received
0927 13:46:08.683572 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:08.683584 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:08.683901 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:08.684181 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:08.685494 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:08.686696 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:08.688352 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:08.688983 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:08.689171 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:08.693143 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:08.693150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:08.693163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:08.693205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:08.696979 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:08.697668 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:08.697678 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:08.697688 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:08.697891 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:08.698300 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:08.698307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:08.698313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:08.698943 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:08.699131 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:08.699949 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 13:46:08.700400 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
I20260927 13:46:08.704229 16572 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:08.704532 16572 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:08.705287 16572 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:08.707374 16387 ts_manager.cc:194] Registered new tserver with Master: 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683)
I20260927 13:46:08.708808 16387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:38335
W20260927 13:46:08.998600 16576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:08.998857 16576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:08.999028 16576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:09.005986 16576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:09.006101 16576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:09.006270 16576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:46:09.015378 16576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:34819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=42889
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:09.016638 16576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:09.017905 16576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:09.030516 16582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:09.030529 16581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:09.030565 16584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:09.031479 16576 server_base.cc:1034] running on GCE node
I20260927 13:46:09.032088 16576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:09.032920 16576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:09.034132 16576 hybrid_clock.cc:648] HybridClock initialized: now 1790516769034089 us; error 54 us; skew 500 ppm
I20260927 13:46:09.034438 16576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:09 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516769, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:09.037384 16576 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:46:09.038635 16576 webserver.cc:460] Webserver started at http://127.1.237.194:42889/ using document root <none> and password file <none>
I20260927 13:46:09.039273 16576 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:09.039386 16576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:09.044265 16576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:46:09.046828 16591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.048157 16576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:46:09.048333 16576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "7e8254dccc0d4268a38028e118c73aa4"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "4a7077ee1d5dc10c7bb6525d2a660de3"
server_key_iv: "4347955d1a0bc3988430b18a20e115bf"
server_key_version: "encryptionkey@0"
I20260927 13:46:09.049238 16576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:09.076407 16576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:09.080647 16576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:09.080900 16576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:09.081825 16576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:09.083086 16576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:09.083214 16576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.083351 16576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:09.083437 16576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.116020 16576 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:34819
I20260927 13:46:09.116104 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:34819 every 8 connection(s)
I20260927 13:46:09.117410 16576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:46:09.121107 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16576
I20260927 13:46:09.121807 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:41513
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=43667
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:46:09 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516769, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:09.140128 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.119859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:58681 (local address 127.1.237.254:43241)
0927 13:46:09.120028 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.120032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.121304 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 13:46:09.124058 (+ 2754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.124062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.124129 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.124212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:09.126140 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.126809 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.129203 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.129641 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.129740 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.133391 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:09.133400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:09.133407 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:09.133452 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:09.137057 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.137702 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.137710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.137715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.137875 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.138244 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.138250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.138255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.138617 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:09.138795 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.139400 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.139666 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:46:09.140632 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.120327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:58681)
0927 13:46:09.121132 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.121181 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.123759 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.124464 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.124478 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.125171 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:09.125958 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.125987 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.126967 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.126975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.129058 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.129076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.129853 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.129860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.130175 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.130883 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:09.130910 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:09.133191 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:09.137177 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:09.137183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:09.137193 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:09.137574 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:09.137974 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:09.137977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:09.137980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:09.138132 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:09.138732 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:09.138738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:09.139012 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.139258 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.139605 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":280,"threads_started":1}
I20260927 13:46:09.142355 16704 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:09.142704 16704 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:09.143666 16704 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:09.145257 16387 ts_manager.cc:194] Registered new tserver with Master: 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
I20260927 13:46:09.146068 16387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:58681
W20260927 13:46:09.437525 16707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:09.437808 16707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:09.437964 16707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:09.446947 16707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:09.447077 16707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:09.447266 16707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:46:09.456904 16707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=43667
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:09.458148 16707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:09.459510 16707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:09.472072 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:09.472121 16713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:09.472025 16712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:09.476528 16707 server_base.cc:1034] running on GCE node
I20260927 13:46:09.477082 16707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:09.477924 16707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:09.479131 16707 hybrid_clock.cc:648] HybridClock initialized: now 1790516769479085 us; error 42 us; skew 500 ppm
I20260927 13:46:09.479460 16707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:09 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516769, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:09.482381 16707 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:46:09.483762 16707 webserver.cc:460] Webserver started at http://127.1.237.195:43667/ using document root <none> and password file <none>
I20260927 13:46:09.484436 16707 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:09.484577 16707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:09.489770 16707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:46:09.492388 16722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.493577 16707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:46:09.493736 16707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "da668c284b3a4420eb7cf9350378e586"
server_key_iv: "966c5a8ce5c2bff2cf1bfd8b903e1aa5"
server_key_version: "encryptionkey@0"
I20260927 13:46:09.494624 16707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:09.510353 16707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:09.514201 16707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:09.514453 16707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:09.515341 16707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:09.516551 16707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:09.516662 16707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.516783 16707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:09.516872 16707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:09.549266 16707 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:41513
I20260927 13:46:09.549340 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:41513 every 8 connection(s)
I20260927 13:46:09.550645 16707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:46:09.556802 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16707
Sep 27 13:46:09 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516769, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:09.570593 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.553253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:46639 (local address 127.1.237.254:43241)
0927 13:46:09.553459 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.553463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.554628 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 13:46:09.556499 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.556504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.556592 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.556693 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:09.558477 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.559398 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.560980 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.561464 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.561591 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.564964 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:09.564969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:09.564973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:09.565001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:09.567709 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.568358 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.568364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.568367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.568510 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.568930 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.568935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.568938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.569189 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:09.569381 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.570031 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.570249 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"spinlock_wait_cycles":1408}
I20260927 13:46:09.571565 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.557681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.1.237.254:43241)
0927 13:46:09.558832 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.558836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.558848 (+ 12us) server_negotiation.cc:407] Connection header received
0927 13:46:09.558886 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.558889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.558934 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.559039 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:09.560503 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.561231 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.562754 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.563230 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.563367 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.566042 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:09.566047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:09.566050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:09.566090 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:09.568871 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.569493 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.569498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.569506 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.569607 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.569963 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.569968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.569971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.570234 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:09.570497 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.571046 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.571287 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":944,"threads_started":1}
I20260927 13:46:09.571699 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.553805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:46639)
0927 13:46:09.554473 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.554513 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.556240 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.556991 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.556999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.557542 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:09.558282 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.558309 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.559555 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.559565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.560838 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.560851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.561688 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.561695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.561936 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.562643 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:09.562664 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:09.564754 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:09.567840 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:09.567848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:09.567863 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:09.568246 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:09.568619 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:09.568625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:09.568629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:09.568815 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:09.569358 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:09.569365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:09.569675 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.569933 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.570416 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
I20260927 13:46:09.573230 16835 heartbeater.cc:347] Connected to a master server at 127.1.237.254:43241
I20260927 13:46:09.573553 16835 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:09.574355 16835 heartbeater.cc:511] Master 127.1.237.254:43241 requested a full tablet report, sending...
I20260927 13:46:09.575852 16387 ts_manager.cc:194] Registered new tserver with Master: 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513)
I20260927 13:46:09.576725 16387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:46639
I20260927 13:46:09.586017 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:46:09.592548 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.586827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.1.237.254:43241)
0927 13:46:09.586968 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.586972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.587504 (+ 532us) server_negotiation.cc:407] Connection header received
0927 13:46:09.587661 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.587664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.587792 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.587901 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:09.588313 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.588978 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.590436 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.590889 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.590986 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.591624 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:09.592080 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:46:09.592200 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.592285 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.592349 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 13:46:09.593413 16387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:46:09.600978 16387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:46:09.623724 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.612233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.1.237.195:41513)
0927 13:46:09.612845 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.612855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.614009 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 13:46:09.614297 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.614305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.614547 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.614730 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:09.615895 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.617997 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.620940 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.622064 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.622201 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.622440 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.623216 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.623374 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":309,"threads_started":1}
I20260927 13:46:09.623991 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.612029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:41513 (local address 127.0.0.1:36642)
0927 13:46:09.613870 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.613899 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.614084 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.614895 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.614902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.614938 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:09.615592 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.615610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.618226 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.618235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.620781 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.620796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.622644 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.622651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.622860 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.622920 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.623134 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.623341 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"mutex_wait_us":3,"thread_start_us":2441,"threads_started":1}
I20260927 13:46:09.626672 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.612357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50474)
0927 13:46:09.614735 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.614750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.618081 (+ 3331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.618993 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.619000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.619028 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:09.619645 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.619667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.621299 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.621305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.623859 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.623874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.625849 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.625855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.626027 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.626078 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.626165 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.626300 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"mutex_wait_us":314,"thread_start_us":1204,"threads_started":1}
I20260927 13:46:09.626765 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.612955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.1.237.193:33683)
0927 13:46:09.613511 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.613521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.618039 (+ 4518us) server_negotiation.cc:407] Connection header received
0927 13:46:09.618359 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.618368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.618601 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.618766 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:09.619829 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.621176 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.624065 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.625555 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.625720 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.625911 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.626250 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.626420 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
I20260927 13:46:09.627570 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.614957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:34819 (local address 127.0.0.1:53260)
0927 13:46:09.618620 (+ 3663us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.618638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.618882 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.619488 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.619493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.619518 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:09.619978 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.619986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.621408 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.621414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.623881 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.623889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.626086 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.626090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.626285 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.626314 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.626398 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.626506 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3552,"mutex_wait_us":118,"thread_start_us":3395,"threads_started":1}
I20260927 13:46:09.628378 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.615362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.1.237.194:34819)
0927 13:46:09.615869 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.615879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.618746 (+ 2867us) server_negotiation.cc:407] Connection header received
0927 13:46:09.619053 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.619062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.619299 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.619484 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:09.620123 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.621299 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.624116 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.625567 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.625720 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.625941 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.626775 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.628037 (+ 1262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":347,"threads_started":1}
I20260927 13:46:09.628198 16770 tablet_service.cc:1467] Processing CreateTablet for tablet 8124e5bd80e34577a8c637276f00f877 (DEFAULT_TABLE table=test-table [id=445243eb592745c3bb5e85d9cbc182cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:09.629642 16770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8124e5bd80e34577a8c637276f00f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:09.630204 16507 tablet_service.cc:1467] Processing CreateTablet for tablet 8124e5bd80e34577a8c637276f00f877 (DEFAULT_TABLE table=test-table [id=445243eb592745c3bb5e85d9cbc182cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:09.631506 16507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8124e5bd80e34577a8c637276f00f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:09.631686 16639 tablet_service.cc:1467] Processing CreateTablet for tablet 8124e5bd80e34577a8c637276f00f877 (DEFAULT_TABLE table=test-table [id=445243eb592745c3bb5e85d9cbc182cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:09.632954 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8124e5bd80e34577a8c637276f00f877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:09.641480 16851 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Bootstrap starting.
I20260927 13:46:09.644519 16851 tablet_bootstrap.cc:654] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:09.645733 16851 log.cc:824] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:09.646996 16852 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Bootstrap starting.
I20260927 13:46:09.648420 16854 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Bootstrap starting.
I20260927 13:46:09.648717 16851 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: No bootstrap required, opened a new log
I20260927 13:46:09.648965 16851 ts_tablet_manager.cc:1397] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 13:46:09.650329 16852 tablet_bootstrap.cc:654] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:09.650713 16854 tablet_bootstrap.cc:654] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:09.651497 16852 log.cc:824] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:09.651731 16854 log.cc:824] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:09.654301 16852 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: No bootstrap required, opened a new log
I20260927 13:46:09.654542 16852 ts_tablet_manager.cc:1397] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 13:46:09.654664 16854 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: No bootstrap required, opened a new log
I20260927 13:46:09.654911 16854 ts_tablet_manager.cc:1397] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 13:46:09.657527 16851 raft_consensus.cc:348] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.657848 16851 raft_consensus.cc:374] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:09.657966 16851 raft_consensus.cc:729] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00da0cbc1d8d4bd5aa7a85aeb01c2228, State: Initialized, Role: FOLLOWER
I20260927 13:46:09.658541 16851 consensus_queue.cc:260] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.659525 16835 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
I20260927 13:46:09.660154 16851 ts_tablet_manager.cc:1428] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 13:46:09.664888 16852 raft_consensus.cc:348] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.664888 16854 raft_consensus.cc:348] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.665277 16854 raft_consensus.cc:374] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:09.665288 16852 raft_consensus.cc:374] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:09.665385 16852 raft_consensus.cc:729] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3317559ec8d04297b624d98034b1d5c6, State: Initialized, Role: FOLLOWER
I20260927 13:46:09.665381 16854 raft_consensus.cc:729] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8254dccc0d4268a38028e118c73aa4, State: Initialized, Role: FOLLOWER
I20260927 13:46:09.665995 16854 consensus_queue.cc:260] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.665997 16852 consensus_queue.cc:260] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.667024 16704 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
I20260927 13:46:09.667747 16854 ts_tablet_manager.cc:1428] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 13:46:09.667975 16852 ts_tablet_manager.cc:1428] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 13:46:09.668220 16572 heartbeater.cc:503] Master 127.1.237.254:43241 was elected leader, sending a full tablet report...
W20260927 13:46:09.679819 16573 tablet.cc:2369] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:46:09.802335 16836 tablet.cc:2369] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:46:09.852662 16859 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:09.852974 16859 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.854888 16859 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
I20260927 13:46:09.861788 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.855477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:51445 (local address 127.1.237.194:34819)
0927 13:46:09.855708 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.855714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.856359 (+ 645us) server_negotiation.cc:407] Connection header received
0927 13:46:09.856528 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.856535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.856620 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.856721 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:09.857385 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.858049 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.860138 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.860878 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.861031 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.861118 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.861423 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.861541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:46:09.861796 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.855312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:44517 (local address 127.1.237.195:41513)
0927 13:46:09.855866 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.855872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.856009 (+ 137us) server_negotiation.cc:407] Connection header received
0927 13:46:09.856230 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.856237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.856340 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.856439 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:09.857128 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.858113 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.860158 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.860887 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.861061 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.861131 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.861445 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.861557 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260927 13:46:09.861941 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.855094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:41513 (local address 127.1.237.193:44517)
0927 13:46:09.855878 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.855897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.856071 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.856469 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.856473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.856501 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:09.856976 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.856987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.858263 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.858267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.859998 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.860009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.861125 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.861133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.861282 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.861305 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.861408 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.861528 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":76,"thread_start_us":416,"threads_started":1}
I20260927 13:46:09.862170 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.855391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:34819 (local address 127.1.237.193:51445)
0927 13:46:09.856248 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:09.856265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:09.856395 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:09.856771 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:09.856775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:09.856802 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:09.857188 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.857202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.858167 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.858175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.859994 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:09.860006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.861086 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.861091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:09.861272 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.861301 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:46:09.861370 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 13:46:09.861511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":70,"thread_start_us":479,"threads_started":1}
I20260927 13:46:09.863065 16790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8124e5bd80e34577a8c637276f00f877" candidate_uuid: "3317559ec8d04297b624d98034b1d5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" is_pre_election: true
I20260927 13:46:09.863517 16659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8124e5bd80e34577a8c637276f00f877" candidate_uuid: "3317559ec8d04297b624d98034b1d5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8254dccc0d4268a38028e118c73aa4" is_pre_election: true
I20260927 13:46:09.863613 16790 raft_consensus.cc:2454] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3317559ec8d04297b624d98034b1d5c6 in term 0.
I20260927 13:46:09.863937 16659 raft_consensus.cc:2454] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3317559ec8d04297b624d98034b1d5c6 in term 0.
I20260927 13:46:09.864253 16461 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 3317559ec8d04297b624d98034b1d5c6; no voters:
I20260927 13:46:09.864691 16859 raft_consensus.cc:2781] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:46:09.864837 16859 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:46:09.864949 16859 raft_consensus.cc:3037] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:09.867094 16859 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.867705 16859 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 1 election: Requested vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
I20260927 13:46:09.868181 16659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8124e5bd80e34577a8c637276f00f877" candidate_uuid: "3317559ec8d04297b624d98034b1d5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8254dccc0d4268a38028e118c73aa4"
I20260927 13:46:09.868369 16790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8124e5bd80e34577a8c637276f00f877" candidate_uuid: "3317559ec8d04297b624d98034b1d5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
I20260927 13:46:09.868463 16659 raft_consensus.cc:3037] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:09.868552 16790 raft_consensus.cc:3037] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 0 FOLLOWER]: Advancing to term 1
W20260927 13:46:09.869249 16705 tablet.cc:2369] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:46:09.870986 16790 raft_consensus.cc:2454] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3317559ec8d04297b624d98034b1d5c6 in term 1.
I20260927 13:46:09.871382 16461 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 3317559ec8d04297b624d98034b1d5c6; no voters:
I20260927 13:46:09.871497 16659 raft_consensus.cc:2454] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3317559ec8d04297b624d98034b1d5c6 in term 1.
I20260927 13:46:09.871650 16859 raft_consensus.cc:2781] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:09.872133 16859 raft_consensus.cc:686] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 LEADER]: Becoming Leader. State: Replica: 3317559ec8d04297b624d98034b1d5c6, State: Running, Role: LEADER
I20260927 13:46:09.872604 16859 consensus_queue.cc:237] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:09.877209 16387 catalog_manager.cc:5495] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "3317559ec8d04297b624d98034b1d5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } health_report { overall_health: UNKNOWN } } }
I20260927 13:46:09.917847 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.912509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.1.237.193:33683)
0927 13:46:09.912701 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.912707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.912726 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:46:09.912910 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.912914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.912987 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.913131 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:09.913500 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.914176 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.915801 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.916208 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.916336 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.916816 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:09.917290 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:46:09.917453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.917512 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.917589 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:46:09.925984 16790 raft_consensus.cc:1261] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Refusing update from remote peer 3317559ec8d04297b624d98034b1d5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:09.925984 16659 raft_consensus.cc:1261] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Refusing update from remote peer 3317559ec8d04297b624d98034b1d5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:09.927066 16859 consensus_queue.cc:1035] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:09.927434 16862 consensus_queue.cc:1035] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:09.951365 16387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:09.951630 16387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:09.955811 16387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Sending DeleteTablet for 3 replicas of tablet 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:09.956885 16507 tablet_service.cc:1514] Processing DeleteTablet for tablet 8124e5bd80e34577a8c637276f00f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:09 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:50474
I20260927 13:46:09.957041 16770 tablet_service.cc:1514] Processing DeleteTablet for tablet 8124e5bd80e34577a8c637276f00f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:09 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:36642
I20260927 13:46:09.957258 16639 tablet_service.cc:1514] Processing DeleteTablet for tablet 8124e5bd80e34577a8c637276f00f877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:09 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53260
I20260927 13:46:09.957870 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16424
I20260927 13:46:09.967098 16876 tablet_replica.cc:330] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: stopping tablet replica
I20260927 13:46:09.967562 16876 raft_consensus.cc:2229] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:09.968081 16876 raft_consensus.cc:2258] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:09.970391 16876 ts_tablet_manager.cc:1905] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:09.975180 16875 tablet_replica.cc:330] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: stopping tablet replica
I20260927 13:46:09.975584 16875 raft_consensus.cc:2229] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:46:09.976312 16875 raft_consensus.cc:2258] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:09.976578 16877 tablet_replica.cc:330] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: stopping tablet replica
I20260927 13:46:09.978223 16877 raft_consensus.cc:2229] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:09.978924 16877 raft_consensus.cc:2258] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:09.979570 16876 ts_tablet_manager.cc:1918] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:09.979722 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:09.959004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.1.237.193:33683)
0927 13:46:09.959226 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:09.959235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:09.959257 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:46:09.959337 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:09.959345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:09.959448 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:09.959576 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:09.961688 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.963314 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.965943 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:09.966978 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:09.967180 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:09.970596 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:09.970612 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:09.970633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:09.970684 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:09.975719 (+ 5035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.976673 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.976680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.976683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.976847 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:09.977303 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:09.977310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:09.977314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:09.977661 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:09.977989 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:09.979101 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 13:46:09.979359 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:46:09.979748 16876 log.cc:1196] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/8124e5bd80e34577a8c637276f00f877
I20260927 13:46:09.980334 16876 ts_tablet_manager.cc:1939] T 8124e5bd80e34577a8c637276f00f877 P 00da0cbc1d8d4bd5aa7a85aeb01c2228: Deleting consensus metadata
I20260927 13:46:09.980356 16567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:09.980965 16877 ts_tablet_manager.cc:1905] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:09.982484 16372 catalog_manager.cc:4841] TS 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): tablet 8124e5bd80e34577a8c637276f00f877 (table test-table [id=445243eb592745c3bb5e85d9cbc182cb]) successfully deleted
I20260927 13:46:09.987449 16877 ts_tablet_manager.cc:1918] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:09.987598 16877 log.cc:1196] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/8124e5bd80e34577a8c637276f00f877
I20260927 13:46:09.988161 16877 ts_tablet_manager.cc:1939] T 8124e5bd80e34577a8c637276f00f877 P 7e8254dccc0d4268a38028e118c73aa4: Deleting consensus metadata
I20260927 13:46:09.989581 16373 catalog_manager.cc:4841] TS 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): tablet 8124e5bd80e34577a8c637276f00f877 (table test-table [id=445243eb592745c3bb5e85d9cbc182cb]) successfully deleted
I20260927 13:46:10.078249 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16424
W20260927 13:46:10.096154 16373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:33683 (error 108)
I20260927 13:46:10.096474 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16576
I20260927 13:46:10.097894 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.097506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50500)
0927 13:46:10.097674 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.097763 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:46:10.098214 16373 catalog_manager.cc:4584] TS 3317559ec8d04297b624d98034b1d5c6 (127.1.237.193:33683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8124e5bd80e34577a8c637276f00f877: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
I20260927 13:46:10.113534 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.097260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.1.237.194:34819)
0927 13:46:10.097436 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:10.097443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:10.097493 (+ 50us) server_negotiation.cc:407] Connection header received
0927 13:46:10.097741 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:10.097746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:10.097836 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:10.097940 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:10.099482 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:10.100199 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:10.101442 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:10.101855 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:10.101983 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:10.108449 (+ 6466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:10.108461 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:10.108474 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:10.108506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:10.110926 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:10.111679 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:10.111688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:10.111696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:10.111813 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:10.112176 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:10.112181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:10.112185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:10.112512 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:10.112808 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:10.113031 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 13:46:10.113233 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:46:10.114115 16699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:10.151614 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.151073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50510)
0927 13:46:10.151330 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.151429 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:46:10.152613 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.152229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50512)
0927 13:46:10.152382 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.152477 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 13:46:10.202023 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16576
I20260927 13:46:10.213259 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.212800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50516)
0927 13:46:10.213011 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.213103 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 13:46:10.214459 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.214140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50530)
0927 13:46:10.214262 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.214354 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 13:46:10.219784 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16707
I20260927 13:46:10.238581 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.220572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.1.237.195:41513)
0927 13:46:10.220744 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:10.220752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:10.220771 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:46:10.220869 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:10.220874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:10.220961 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:10.221069 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:10.222333 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:10.223431 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:10.224680 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:10.225278 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:10.225405 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:10.233679 (+ 8274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:10.233696 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:10.233712 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:10.233743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:10.236330 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:10.236939 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:10.236945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:10.236947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:10.237075 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:10.237362 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:10.237365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:10.237367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:10.237672 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:10.237888 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:10.238093 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 13:46:10.238236 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:46:10.239249 16830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:10.295018 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.294487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50542)
0927 13:46:10.294705 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.294823 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 13:46:10.296105 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:10.295683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:33683 (local address 127.0.0.1:50556)
0927 13:46:10.295889 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:10.295973 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 13:46:10.326097 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16707
I20260927 13:46:10.341773 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16354
I20260927 13:46:10.342832 16417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:10.452066 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16354
I20260927 13:46:10.473181 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--webserver_interface=127.1.237.254
--webserver_port=46217
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:10.784293 16883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:10.784580 16883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:10.784731 16883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:10.791927 16883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:10.792024 16883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:10.792132 16883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:10.792208 16883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:10.792272 16883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:10.801838 16883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:43241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=46217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:10.803140 16883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:10.804459 16883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:10.815536 16888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:10.815625 16889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:10.815577 16891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:10.816812 16883 server_base.cc:1034] running on GCE node
I20260927 13:46:10.817579 16883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:10.818943 16883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:10.820185 16883 hybrid_clock.cc:648] HybridClock initialized: now 1790516770820116 us; error 76 us; skew 500 ppm
I20260927 13:46:10.820526 16883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:10 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516770, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:10.823614 16883 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:10.825183 16883 webserver.cc:460] Webserver started at http://127.1.237.254:46217/ using document root <none> and password file <none>
I20260927 13:46:10.825877 16883 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:10.826025 16883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:10.831429 16883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 13:46:10.834129 16898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:10.835500 16883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:46:10.835680 16883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "2b013e8a7f7a4bf2b6adaadce559b71f"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "b8696c7b45e6638a9bcb4bc2cb755826"
server_key_iv: "7a3f635d634572a502300939b939c316"
server_key_version: "encryptionkey@0"
I20260927 13:46:10.836635 16883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:10.852092 16883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:10.855836 16883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:10.856087 16883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:10.873795 16883 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:43241
I20260927 13:46:10.873816 16949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:43241 every 8 connection(s)
I20260927 13:46:10.875293 16883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:10.881719 16950 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:46:10.883496 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16883
I20260927 13:46:10.884586 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:33683
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=40347
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:46:10.886760 16950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap starting.
I20260927 13:46:10.907897 16950 log.cc:824] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:10.923197 16950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:46:10.923723 16950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f: Bootstrap complete.
I20260927 13:46:10.932608 16950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:10.933458 16950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Initialized, Role: FOLLOWER
I20260927 13:46:10.934003 16950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:10.934263 16950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:10.934381 16950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:10.934554 16950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:46:10.938161 16950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:10.938992 16950 leader_election.cc:302] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b013e8a7f7a4bf2b6adaadce559b71f; no voters:
I20260927 13:46:10.939666 16950 leader_election.cc:288] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 13:46:10.939801 16954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:46:10.941912 16950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:10.943419 16954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [term 3 LEADER]: Becoming Leader. State: Replica: 2b013e8a7f7a4bf2b6adaadce559b71f, State: Running, Role: LEADER
I20260927 13:46:10.944092 16954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } }
I20260927 13:46:10.946612 16956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b013e8a7f7a4bf2b6adaadce559b71f. Latest consensus state: current_term: 3 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:10.947008 16956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:10.947220 16955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b013e8a7f7a4bf2b6adaadce559b71f" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43241 } } }
I20260927 13:46:10.947584 16955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b013e8a7f7a4bf2b6adaadce559b71f [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:10.960053 16963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:10.968525 16963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:10.970144 16963 catalog_manager.cc:1259] Loaded cluster ID: ce13067eba6f4f31b93be6b3014b466f
I20260927 13:46:10.970233 16963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 13:46:10.972291 16963 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f37c544c9ad google::LogMessage::Fail() at ??:0
@ 0x7f37c544f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f37c544c4de google::LogMessage::Flush() at ??:0
@ 0x7f37c544da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f37dae83512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f37dae80976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f37daf08610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f37daf08439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f37c8098345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f37c809b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f37c809b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f37c8076eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f37cb3506db start_thread at ??:0
@ 0x7f37c42e661f clone at ??:0
W20260927 13:46:11.283406 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:11.283653 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:11.283742 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:11.290555 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:11.290645 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:11.290748 16952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:46:11.300050 16952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:33683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=40347
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:11.301167 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:11.302357 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:11.315044 16994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:11.315097 16991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:11.315125 16992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:11.315910 16952 server_base.cc:1034] running on GCE node
I20260927 13:46:11.316494 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:11.317427 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:11.318653 16952 hybrid_clock.cc:648] HybridClock initialized: now 1790516771318591 us; error 52 us; skew 500 ppm
I20260927 13:46:11.318944 16952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:11 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:11.321945 16952 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:46:11.323414 16952 webserver.cc:460] Webserver started at http://127.1.237.193:40347/ using document root <none> and password file <none>
I20260927 13:46:11.324055 16952 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:11.324195 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:11.329291 16952 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:46:11.332046 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:11.333313 16952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:11.333487 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "3317559ec8d04297b624d98034b1d5c6"
format_stamp: "Formatted at 2026-09-27 13:46:05 on dist-test-slave-7g3l"
server_key: "594ac244bbd747c2ac873b8b75ae5d05"
server_key_iv: "b1358b470bc9ca32879ea9aec0e60eeb"
server_key_version: "encryptionkey@0"
I20260927 13:46:11.334400 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:11.358428 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:11.362154 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:11.362437 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:11.363430 16952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:11.365154 17008 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 13:46:11.370569 16952 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 13:46:11.370709 16952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 13:46:11.370857 16952 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 13:46:11.373904 16952 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 13:46:11.374004 16952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260927 13:46:11.374343 17008 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Bootstrap starting.
I20260927 13:46:11.383700 17008 log.cc:824] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:11.401216 17008 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:46:11.402002 17008 tablet_bootstrap.cc:492] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Bootstrap complete.
I20260927 13:46:11.402686 17008 ts_tablet_manager.cc:1397] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20260927 13:46:11.409858 16952 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:33683
I20260927 13:46:11.409889 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:33683 every 8 connection(s)
I20260927 13:46:11.411437 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:46:11.414326 17008 raft_consensus.cc:348] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:11.415725 17008 raft_consensus.cc:729] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3317559ec8d04297b624d98034b1d5c6, State: Initialized, Role: FOLLOWER
I20260927 13:46:11.416324 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.414254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:45953)
0927 13:46:11.415093 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.415397 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":366,"threads_started":1}
I20260927 13:46:11.416347 17008 consensus_queue.cc:260] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
W20260927 13:46:11.416934 17116 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:43241 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:43241: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
I20260927 13:46:11.417728 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.417415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:52583)
0927 13:46:11.417551 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.417623 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 13:46:11.417830 17008 ts_tablet_manager.cc:1428] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 13:46:11.418507 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.418183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:59897)
0927 13:46:11.418337 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.418402 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:46:11.418733 17116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:46:11.419348 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 16952
I20260927 13:46:11.419436 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.419123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:59645)
0927 13:46:11.419286 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.419342 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 13:46:11.420018 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:34819
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=42889
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:11.731572 17120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:11.731861 17120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:11.732031 17120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:11.739059 17120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:11.739204 17120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:11.739406 17120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:46:11.748499 17120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:34819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=42889
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:11.749691 17120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:11.750932 17120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:11.763989 17125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:11.763916 17126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:11.764038 17128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:11.764678 17120 server_base.cc:1034] running on GCE node
I20260927 13:46:11.765237 17120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:11.766065 17120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:11.767292 17120 hybrid_clock.cc:648] HybridClock initialized: now 1790516771767238 us; error 42 us; skew 500 ppm
I20260927 13:46:11.767551 17120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:11 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:11.770539 17120 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:46:11.772217 17120 webserver.cc:460] Webserver started at http://127.1.237.194:42889/ using document root <none> and password file <none>
I20260927 13:46:11.772878 17120 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:11.773003 17120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:11.777997 17120 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 13:46:11.780623 17135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:11.781822 17120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:46:11.781988 17120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "7e8254dccc0d4268a38028e118c73aa4"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "4a7077ee1d5dc10c7bb6525d2a660de3"
server_key_iv: "4347955d1a0bc3988430b18a20e115bf"
server_key_version: "encryptionkey@0"
I20260927 13:46:11.782907 17120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:11.796928 17120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:11.800855 17120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:11.801117 17120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:11.802075 17120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:11.803324 17120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:11.803416 17120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:11.803570 17120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:11.803659 17120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:11.836884 17120 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:34819
I20260927 13:46:11.836954 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:34819 every 8 connection(s)
I20260927 13:46:11.838413 17120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:46:11.839775 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17120
I20260927 13:46:11.840469 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:41513
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:43241
--webserver_port=43667
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:46:11.843530 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.841606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:47873)
0927 13:46:11.842393 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.842610 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
W20260927 13:46:11.844298 17248 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:43241 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:43241: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
I20260927 13:46:11.845192 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.844859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:51665)
0927 13:46:11.845023 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.845094 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 13:46:11.846132 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:11.845809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:35361)
0927 13:46:11.845959 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:11.846029 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 13:46:11.846410 17248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 13:46:12.153427 17251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:12.153663 17251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:12.153836 17251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:12.161092 17251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:12.161185 17251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:12.161370 17251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:46:12.170482 17251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=43667
--tserver_master_addrs=127.1.237.254:43241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:12.171742 17251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:12.172945 17251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:12.185256 17259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:12.185341 17256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:12.185252 17257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:12.185828 17251 server_base.cc:1034] running on GCE node
I20260927 13:46:12.186606 17251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:12.187639 17251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:12.188854 17251 hybrid_clock.cc:648] HybridClock initialized: now 1790516772188812 us; error 52 us; skew 500 ppm
I20260927 13:46:12.189150 17251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:12 dist-test-slave-7g3l krb5kdc[15803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:12.192095 17251 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:46:12.193449 17251 webserver.cc:460] Webserver started at http://127.1.237.195:43667/ using document root <none> and password file <none>
I20260927 13:46:12.194069 17251 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:12.194202 17251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:12.199072 17251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:12.201663 17266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:12.202912 17251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:46:12.203074 17251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228"
format_stamp: "Formatted at 2026-09-27 13:46:06 on dist-test-slave-7g3l"
server_key: "da668c284b3a4420eb7cf9350378e586"
server_key_iv: "966c5a8ce5c2bff2cf1bfd8b903e1aa5"
server_key_version: "encryptionkey@0"
I20260927 13:46:12.204061 17251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:12.228152 17251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:12.231735 17251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:12.232003 17251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:12.232899 17251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:12.234166 17251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:12.234282 17251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:12.234411 17251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:12.234504 17251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:12.266598 17251 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:41513
I20260927 13:46:12.266676 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:41513 every 8 connection(s)
I20260927 13:46:12.268062 17251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:46:12.272557 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:12.270818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:37089)
0927 13:46:12.271560 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:12.271796 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":332,"threads_started":1}
W20260927 13:46:12.273259 17379 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:43241 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:43241: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
I20260927 13:46:12.273337 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17251
I20260927 13:46:12.274026 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:12.273729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:54747)
0927 13:46:12.273876 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:12.273939 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 13:46:12.274755 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:12.274466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:44283)
0927 13:46:12.274588 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:12.274660 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:46:12.275031 17379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:46:12.420818 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:12.420114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:53277)
0927 13:46:12.420542 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:12.420620 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":226,"threads_started":1}
I20260927 13:46:12.847805 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:12.846951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:58103)
0927 13:46:12.847480 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:12.847599 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":263,"threads_started":1}
I20260927 13:46:13.274955 17385 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:13.275254 17385 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:13.276262 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.275573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:55757)
0927 13:46:13.275987 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:13.276063 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260927 13:46:13.276975 17385 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
Sep 27 13:46:13 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
Sep 27 13:46:13 dist-test-slave-7g3l krb5kdc[15803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:46:13.309338 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.277364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:46207 (local address 127.1.237.195:41513)
0927 13:46:13.287966 (+ 10602us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:13.287988 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:13.288743 (+ 755us) server_negotiation.cc:407] Connection header received
0927 13:46:13.290786 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:13.290793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:13.291134 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:13.291572 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:13.293782 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.294978 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.296777 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.297384 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.297570 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:13.301608 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:13.301619 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:13.301634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:13.301675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:13.305139 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:13.306352 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:13.306361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:13.306369 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:13.306563 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:13.307018 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:13.307022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:13.307024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:13.307585 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:13.307732 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:13.308574 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 13:46:13.308946 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10431,"thread_start_us":10291,"threads_started":1}
I20260927 13:46:13.309347 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.277421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:34819 (local address 127.1.237.193:50045)
0927 13:46:13.289163 (+ 11742us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:13.289177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:13.289327 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:13.291710 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:13.291715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:13.292890 (+ 1175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:13.293609 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:13.293627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.295077 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.295081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:13.296638 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:13.296649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.297625 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.297631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:13.297882 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:13.298651 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:13.298678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:13.301074 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:13.305297 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:13.305306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:13.305321 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:13.306176 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:13.306996 (+ 820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:13.306999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:13.307001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:13.307101 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:13.308052 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:13.308057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:13.308187 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 13:46:13.308357 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 13:46:13.308851 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11629,"mutex_wait_us":145,"spinlock_wait_cycles":896,"thread_start_us":11483,"threads_started":1}
I20260927 13:46:13.309405 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.277221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:41513 (local address 127.1.237.193:46207)
0927 13:46:13.288312 (+ 11091us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:13.288363 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:13.289055 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:13.291692 (+ 2637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:13.291703 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:13.292908 (+ 1205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:13.293610 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:13.293628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.295235 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.295239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:13.296644 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:13.296656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.297583 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.297593 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:13.297882 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:13.298591 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:13.298614 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:13.301373 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:13.305297 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:13.305306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:13.305321 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:13.306176 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:13.306709 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:13.306715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:13.306718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:13.306899 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:13.307715 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:13.307729 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:13.308071 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 13:46:13.308357 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 13:46:13.308997 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10880,"mutex_wait_us":310,"spinlock_wait_cycles":28800,"thread_start_us":10754,"threads_started":1}
I20260927 13:46:13.309401 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.277611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:50045 (local address 127.1.237.194:34819)
0927 13:46:13.288784 (+ 11173us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:13.288799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:13.289294 (+ 495us) server_negotiation.cc:407] Connection header received
0927 13:46:13.290957 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:13.290965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:13.291423 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:13.291804 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:13.293804 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.294905 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.296789 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:13.297393 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:13.297583 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:13.301356 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:13.301363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:13.301375 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:13.301407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:13.305139 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:13.306352 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:13.306361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:13.306369 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:13.306559 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:13.307241 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:13.307246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:13.307249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:13.307896 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:13.308079 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:13.308583 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 13:46:13.308978 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11000,"thread_start_us":10847,"threads_started":1}
W20260927 13:46:13.312383 17003 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
W20260927 13:46:13.312727 17004 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:13.312867 17004 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6; no voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 7e8254dccc0d4268a38028e118c73aa4
I20260927 13:46:13.313206 17385 raft_consensus.cc:2726] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:46:13.422021 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.421652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:33991)
0927 13:46:13.421803 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:13.421899 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 13:46:13.849403 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:13.848725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:53603)
0927 13:46:13.849157 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:13.849233 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":243,"threads_started":1}
I20260927 13:46:14.277640 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:14.277029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:56575)
0927 13:46:14.277428 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:14.277500 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":220,"threads_started":1}
I20260927 13:46:14.423645 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:14.422901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:45069)
0927 13:46:14.423401 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:14.423487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":281,"threads_started":1}
I20260927 13:46:14.850916 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:14.850249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:59427)
0927 13:46:14.850655 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:14.850738 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":225,"threads_started":1}
I20260927 13:46:14.918780 17395 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:14.918970 17395 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:14.919615 17395 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
W20260927 13:46:14.920564 17003 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
W20260927 13:46:14.920765 17004 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:14.920897 17004 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6; no voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 7e8254dccc0d4268a38028e118c73aa4
I20260927 13:46:14.921155 17395 raft_consensus.cc:2726] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:46:15.279011 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:15.278393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:60797)
0927 13:46:15.278789 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:15.278863 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":212,"threads_started":1}
I20260927 13:46:15.425120 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:15.424466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:49233)
0927 13:46:15.424876 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:15.424955 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":183,"threads_started":1}
I20260927 13:46:15.852459 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:15.851799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:37241)
0927 13:46:15.852203 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:15.852281 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":196,"threads_started":1}
I20260927 13:46:16.280592 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:16.279922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:59879)
0927 13:46:16.280335 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:16.280412 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260927 13:46:16.426707 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:16.426075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:48425)
0927 13:46:16.426483 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:16.426559 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
I20260927 13:46:16.714676 17401 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:16.714906 17401 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:16.715607 17401 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
W20260927 13:46:16.716565 17004 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
W20260927 13:46:16.716761 17003 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:16.716903 17003 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6; no voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 7e8254dccc0d4268a38028e118c73aa4
I20260927 13:46:16.717206 17401 raft_consensus.cc:2726] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:46:16.853928 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:16.853274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:36889)
0927 13:46:16.853691 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:16.853769 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260927 13:46:17.282071 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:17.281396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:41187)
0927 13:46:17.281846 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:17.281920 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260927 13:46:17.428148 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:17.427524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:45681)
0927 13:46:17.427884 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:17.427956 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":190,"threads_started":1}
I20260927 13:46:17.855587 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:17.854856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:55223)
0927 13:46:17.855329 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:17.855412 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260927 13:46:18.283835 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:18.282861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:58459)
0927 13:46:18.283477 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:18.283589 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":242,"threads_started":1}
I20260927 13:46:18.429579 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:18.428908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:47221)
0927 13:46:18.429315 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:18.429392 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":218,"threads_started":1}
I20260927 13:46:18.763296 17408 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:18.763479 17408 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:18.764114 17408 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
W20260927 13:46:18.765043 17003 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
W20260927 13:46:18.765182 17004 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:18.765370 17004 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6; no voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 7e8254dccc0d4268a38028e118c73aa4
I20260927 13:46:18.765614 17408 raft_consensus.cc:2726] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:46:18.857167 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:18.856389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:32793)
0927 13:46:18.856901 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:18.856988 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":243,"threads_started":1}
I20260927 13:46:19.285432 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:19.284740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:51935)
0927 13:46:19.285171 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:19.285252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":204,"threads_started":1}
I20260927 13:46:19.431062 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:19.430402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:34201)
0927 13:46:19.430805 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:19.430899 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":196,"threads_started":1}
I20260927 13:46:19.858820 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:19.858175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:37089)
0927 13:46:19.858572 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:19.858646 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":209,"threads_started":1}
I20260927 13:46:20.287055 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:20.286242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:50459)
0927 13:46:20.286799 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:20.286884 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":360,"threads_started":1}
I20260927 13:46:20.432556 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:20.431862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:55847)
0927 13:46:20.432299 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:20.432375 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
I20260927 13:46:20.860450 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:20.859660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:59527)
0927 13:46:20.860173 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:20.860281 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":282,"threads_started":1}
I20260927 13:46:21.288782 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:21.287996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:60949)
0927 13:46:21.288555 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:21.288636 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260927 13:46:21.434105 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:21.433463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:59955)
0927 13:46:21.433862 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:21.433943 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":202,"threads_started":1}
I20260927 13:46:21.861997 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:21.861316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:34577)
0927 13:46:21.861732 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:21.861816 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":221,"threads_started":1}
I20260927 13:46:22.290213 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:22.289583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:60247)
0927 13:46:22.289977 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:22.290053 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":214,"threads_started":1}
I20260927 13:46:22.435730 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:22.434957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:40781)
0927 13:46:22.435493 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:22.435571 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260927 13:46:22.863629 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:22.862869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:38411)
0927 13:46:22.863377 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:22.863458 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":218,"threads_started":1}
I20260927 13:46:23.291814 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:23.291023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:43619)
0927 13:46:23.291576 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:23.291653 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":323,"threads_started":1}
I20260927 13:46:23.437852 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:23.436721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:56321)
0927 13:46:23.437384 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:23.437536 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":341,"threads_started":1}
I20260927 13:46:23.865156 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:23.864483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:47207)
0927 13:46:23.864915 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:23.864995 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260927 13:46:24.293533 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:24.292807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:37809)
0927 13:46:24.293297 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:24.293373 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
I20260927 13:46:24.439636 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:24.438850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:47701)
0927 13:46:24.439394 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:24.439484 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":317,"threads_started":1}
I20260927 13:46:24.866743 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:24.866002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:43419)
0927 13:46:24.866479 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:24.866586 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260927 13:46:25.295276 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:25.294426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:60211)
0927 13:46:25.294947 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:25.295038 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
I20260927 13:46:25.441509 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:25.440616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:50167)
0927 13:46:25.441118 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:25.441214 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260927 13:46:25.486902 17430 raft_consensus.cc:482] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:25.487138 17430 raft_consensus.cc:504] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00da0cbc1d8d4bd5aa7a85aeb01c2228" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 41513 } } peers { permanent_uuid: "3317559ec8d04297b624d98034b1d5c6" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 33683 } } peers { permanent_uuid: "7e8254dccc0d4268a38028e118c73aa4" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 34819 } }
I20260927 13:46:25.487829 17430 leader_election.cc:288] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513), 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819)
W20260927 13:46:25.488795 17004 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8254dccc0d4268a38028e118c73aa4 (127.1.237.194:34819): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
W20260927 13:46:25.489012 17003 leader_election.cc:341] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00da0cbc1d8d4bd5aa7a85aeb01c2228 (127.1.237.195:41513): Not found: Tablet not found: 8124e5bd80e34577a8c637276f00f877
I20260927 13:46:25.489187 17003 leader_election.cc:302] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3317559ec8d04297b624d98034b1d5c6; no voters: 00da0cbc1d8d4bd5aa7a85aeb01c2228, 7e8254dccc0d4268a38028e118c73aa4
I20260927 13:46:25.489446 17430 raft_consensus.cc:2726] T 8124e5bd80e34577a8c637276f00f877 P 3317559ec8d04297b624d98034b1d5c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 13:46:25.868319 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:25.867628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:40875)
0927 13:46:25.868051 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:25.868127 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260927 13:46:26.296715 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:26.296047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:51133)
0927 13:46:26.296461 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:26.296536 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260927 13:46:26.443346 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:26.442645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.193:45187)
0927 13:46:26.443029 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:26.443113 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":208,"threads_started":1}
I20260927 13:46:26.869946 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:26.869262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.194:53427)
0927 13:46:26.869694 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:26.869790 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":190,"threads_started":1}
I20260927 13:46:27.276098 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 16952
I20260927 13:46:27.290858 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:27.277012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.1.237.193:33683)
0927 13:46:27.277644 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:27.277653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:27.277684 (+ 31us) server_negotiation.cc:407] Connection header received
0927 13:46:27.277754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:27.277762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:27.278041 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:27.278219 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:27.280604 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.281506 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.282685 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.283281 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.283374 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:27.285733 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:27.285744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:27.285756 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:27.285790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:27.288573 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.289194 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.289200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.289202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.289341 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.289654 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.289657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.289659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.289965 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:27.290178 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:27.290426 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 13:46:27.290602 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":309,"threads_started":1}
I20260927 13:46:27.292148 17111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:27.298389 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:27.297717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:43241 (local address 127.1.237.195:45815)
0927 13:46:27.298155 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:27.298248 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260927 13:46:27.377405 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 16952
I20260927 13:46:27.394308 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17120
I20260927 13:46:27.406318 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:27.394975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.1.237.194:34819)
0927 13:46:27.395421 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:27.395425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:27.395441 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:46:27.395495 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:27.395498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:27.395555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:27.395643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:27.396859 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.397596 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.398837 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.399311 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.399406 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:27.401885 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:27.401890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:27.401893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:27.401921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:27.404264 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.404832 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.404835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.404838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.404953 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.405255 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.405258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.405260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.405487 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:27.405677 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:27.405900 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 13:46:27.406043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":254,"threads_started":1}
I20260927 13:46:27.406991 17243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:27.490789 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17120
I20260927 13:46:27.508237 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17251
I20260927 13:46:27.520602 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:27.508970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.1.237.195:41513)
0927 13:46:27.509486 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:27.509490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:27.509507 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:46:27.509558 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:27.509561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:27.509622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:27.509704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:27.510972 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.511765 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.512934 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:27.513321 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:27.513406 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:27.516026 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:27.516031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:27.516034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:27.516062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:27.518411 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.519025 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.519028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.519030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.519147 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:27.519510 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:27.519512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:27.519514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:27.519760 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:27.519943 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:27.520174 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 13:46:27.520334 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":279,"threads_started":1}
I20260927 13:46:27.521234 17374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:27.600884 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17251
2026-09-27T13:46:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24087 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:46:27 dist-test-slave-7g3l krb5kdc[17444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:46:27 dist-test-slave-7g3l krb5kdc[17444](info): set up 2 sockets
Sep 27 13:46:27 dist-test-slave-7g3l krb5kdc[17444](info): commencing operation
krb5kdc: starting...
W20260927 13:46:28.679670 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:46:28 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:46:28.891436 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:29.200157 17457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:29.200464 17457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:29.200627 17457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:29.207307 17457 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:46:29.207424 17457 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:46:29.207536 17457 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:46:29.207618 17457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:29.207702 17457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:29.207823 17457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:29.207911 17457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:29.208002 17457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:29.216893 17457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:29.218012 17457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:29.219233 17457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:29.230042 17463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:29.230073 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:29.230151 17462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:29.230599 17457 server_base.cc:1034] running on GCE node
I20260927 13:46:29.231465 17457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:29.232806 17457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:29.234048 17457 hybrid_clock.cc:648] HybridClock initialized: now 1790516789233982 us; error 54 us; skew 500 ppm
I20260927 13:46:29.234354 17457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:29 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516789, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:29.243276 17457 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:29.244679 17457 webserver.cc:460] Webserver started at http://127.1.237.254:43469/ using document root <none> and password file <none>
I20260927 13:46:29.245371 17457 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:29.245496 17457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:29.245803 17457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:29.248718 17457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "bbf1a8f08b094e99870a01ca425254f5"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "ced75748b00ea41bcaf9b9137ab8011e"
server_key_iv: "d19ecf7d351c42e46a2f21bc743eb375"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.249660 17457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "bbf1a8f08b094e99870a01ca425254f5"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "ced75748b00ea41bcaf9b9137ab8011e"
server_key_iv: "d19ecf7d351c42e46a2f21bc743eb375"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.254767 17457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:46:29.258071 17472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:29.259465 17457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:29.259630 17457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "bbf1a8f08b094e99870a01ca425254f5"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "ced75748b00ea41bcaf9b9137ab8011e"
server_key_iv: "d19ecf7d351c42e46a2f21bc743eb375"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.259840 17457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:29.269831 17457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:29.273525 17457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:29.273773 17457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:29.291682 17457 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46413
I20260927 13:46:29.291728 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46413 every 8 connection(s)
I20260927 13:46:29.293011 17457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:29.296309 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:29.300122 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17457
I20260927 13:46:29.300319 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:46:29.300951 1975 external_mini_cluster.cc:1428] Setting key e4fd7d629a248e31e0d3933950922b34
I20260927 13:46:29.306876 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap starting.
I20260927 13:46:29.310313 17524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:29.311554 17524 log.cc:824] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:29.314774 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: No bootstrap required, opened a new log
Sep 27 13:46:29 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:29.322248 17524 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:29.322652 17524 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:29.322793 17524 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Initialized, Role: FOLLOWER
I20260927 13:46:29.323360 17524 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:29.323606 17524 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:29.323748 17524 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:29.323890 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:29.302799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.1.237.254:46413)
0927 13:46:29.303604 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:29.303614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:29.303653 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:46:29.305443 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:29.305463 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:29.305794 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:29.306069 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:29.307830 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.309103 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.311035 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.311835 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.311952 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:29.316663 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:29.316670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:29.316686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:29.316717 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:29.319837 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:29.320491 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:29.320498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:29.320503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:29.320659 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:29.321032 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:29.321038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:29.321047 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:29.321533 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:29.321809 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:29.322256 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 13:46:29.322678 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":376,"threads_started":1}
I20260927 13:46:29.323920 17524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:29.326090 17524 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:29.326594 17524 leader_election.cc:302] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf1a8f08b094e99870a01ca425254f5; no voters:
I20260927 13:46:29.327055 17524 leader_election.cc:288] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:46:29.327221 17529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:29.327879 17529 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 LEADER]: Becoming Leader. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Running, Role: LEADER
I20260927 13:46:29.328405 17529 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:29.329087 17524 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:29.331243 17530 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:46:29.331312 17531 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf1a8f08b094e99870a01ca425254f5. Latest consensus state: current_term: 1 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:46:29.331637 17530 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:29.331707 17531 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:29.334641 17535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:29.340721 17535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:29.349455 17535 catalog_manager.cc:1347] Generated new cluster ID: b7a511b6ed2f40e7a4411c2be9d8fa34
I20260927 13:46:29.349563 17535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:29.416884 17535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:46:29.418144 17535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:29.427930 17535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Generated new TSK 0
I20260927 13:46:29.428853 17535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:46:29.488276 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:29.819730 17550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:29.819975 17550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:29.820063 17550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:29.826954 17550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:29.827054 17550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:29.827199 17550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:46:29.836735 17550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:29.837893 17550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:29.839129 17550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:29.851502 17556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:29.851576 17558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:29.851620 17555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:29.852710 17550 server_base.cc:1034] running on GCE node
I20260927 13:46:29.853293 17550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:29.854090 17550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:29.855273 17550 hybrid_clock.cc:648] HybridClock initialized: now 1790516789855226 us; error 51 us; skew 500 ppm
I20260927 13:46:29.855593 17550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:29 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516789, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:29.858546 17550 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:46:29.859961 17550 webserver.cc:460] Webserver started at http://127.1.237.193:34973/ using document root <none> and password file <none>
I20260927 13:46:29.860607 17550 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:29.860742 17550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:29.861024 17550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:29.863864 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "1a0bce340fd5dec472c654f34bbfcdb5"
server_key_iv: "46197b74e9121f962a02fcaf278c171c"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.864652 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "1a0bce340fd5dec472c654f34bbfcdb5"
server_key_iv: "46197b74e9121f962a02fcaf278c171c"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.869410 17550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 13:46:29.872537 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:29.873711 17550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 13:46:29.873871 17550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "1a0bce340fd5dec472c654f34bbfcdb5"
server_key_iv: "46197b74e9121f962a02fcaf278c171c"
server_key_version: "encryptionkey@0"
I20260927 13:46:29.874054 17550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:29.895735 17550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:29.899541 17550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:29.899807 17550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:29.900733 17550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:29.901960 17550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:29.902061 17550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:29.902205 17550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:29.902293 17550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:29.935583 17550 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:42217
I20260927 13:46:29.935760 17677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:42217 every 8 connection(s)
I20260927 13:46:29.936873 17550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:46:29.940444 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17550
I20260927 13:46:29.940655 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:46:29.941318 1975 external_mini_cluster.cc:1428] Setting key 3021e41e25fff4ee58ec7ed96195e79f
Sep 27 13:46:29 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516789, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:29.958986 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:29.939289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:38693 (local address 127.1.237.254:46413)
0927 13:46:29.939728 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:29.939732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:29.942742 (+ 3010us) server_negotiation.cc:407] Connection header received
0927 13:46:29.942800 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:29.942804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:29.942870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:29.942978 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:29.944985 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.945716 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.948378 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.948812 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.948920 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:29.953357 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:29.953362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:29.953365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:29.953417 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:29.955972 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:29.956703 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:29.956725 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:29.956727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:29.956838 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:29.957252 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:29.957256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:29.957259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:29.957516 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:29.957695 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:29.958403 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 13:46:29.958632 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260927 13:46:29.959997 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:29.939697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:38693)
0927 13:46:29.940418 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:29.940460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:29.942385 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:29.943274 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:29.943283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:29.943999 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:29.944796 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:29.944823 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.945878 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.945887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:29.948203 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:29.948222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:29.949056 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:29.949066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:29.949390 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:29.950146 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:29.950180 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:29.953086 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:29.956115 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:29.956124 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:29.956141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:29.956562 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:29.956946 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:29.956951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:29.956954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:29.957134 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:29.957684 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:29.957691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:29.958024 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 13:46:29.958292 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 13:46:29.958855 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":286,"threads_started":1}
I20260927 13:46:29.961493 17678 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:29.961824 17678 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:29.962495 17678 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:29.964675 17489 ts_manager.cc:194] Registered new tserver with Master: a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217)
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:46:29.966120 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:38693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:46:29.994247 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:30.307117 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:30.307484 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:30.307680 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:30.314896 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:30.314989 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:30.315196 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:46:30.324733 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:30.325959 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:30.327306 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:30.340278 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:30.340399 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:30.340472 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:30.341189 17684 server_base.cc:1034] running on GCE node
I20260927 13:46:30.341688 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:30.342499 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:30.343717 17684 hybrid_clock.cc:648] HybridClock initialized: now 1790516790343670 us; error 42 us; skew 500 ppm
I20260927 13:46:30.344010 17684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:30 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:30.347013 17684 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:46:30.348412 17684 webserver.cc:460] Webserver started at http://127.1.237.194:33723/ using document root <none> and password file <none>
I20260927 13:46:30.349038 17684 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:30.349179 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:30.349484 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:30.352267 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "bbeb654eae544921bea27e059de97db1"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "2cc462b1177c76140e7a408a6a35f225"
server_key_iv: "ebba4dd25a240571d383b2163b8130c2"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.352996 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "bbeb654eae544921bea27e059de97db1"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "2cc462b1177c76140e7a408a6a35f225"
server_key_iv: "ebba4dd25a240571d383b2163b8130c2"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.357825 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:46:30.361022 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.362174 17684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:46:30.362335 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "bbeb654eae544921bea27e059de97db1"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "2cc462b1177c76140e7a408a6a35f225"
server_key_iv: "ebba4dd25a240571d383b2163b8130c2"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.362516 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:30.380784 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:30.385087 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:30.385382 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:30.386284 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:30.387593 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:30.387692 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.387844 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:30.387916 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.419337 17684 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:38765
I20260927 13:46:30.419356 17811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:38765 every 8 connection(s)
I20260927 13:46:30.420542 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:46:30.424840 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17684
I20260927 13:46:30.425024 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:46:30.425470 1975 external_mini_cluster.cc:1428] Setting key 06ee489b3d565c3e24506aa0401fd80f
Sep 27 13:46:30 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:30.440001 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.422747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:56731 (local address 127.1.237.254:46413)
0927 13:46:30.422937 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.422941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.424039 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 13:46:30.425924 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.425930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.426017 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.426124 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:30.427851 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.428581 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.430226 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.430745 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.430875 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.434190 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:30.434195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:30.434198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:30.434227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:30.437102 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.437770 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.437776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.437779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.437909 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.438321 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.438325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.438329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.438591 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:30.438745 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.439445 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.439669 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:46:30.440887 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.423148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:56731)
0927 13:46:30.423877 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:30.423919 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:30.425667 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:30.426381 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:30.426392 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:30.427072 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:30.427692 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.427708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.428729 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.428734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.430081 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.430092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.430971 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.430978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.431269 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.431922 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:30.431947 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:30.433979 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:30.437233 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:30.437241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:30.437256 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:30.437647 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:30.438017 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:30.438023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:30.438027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:30.438197 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:30.438732 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:30.438739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:30.439043 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 13:46:30.439340 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 13:46:30.439838 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":264,"threads_started":1}
I20260927 13:46:30.442240 17812 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:30.442540 17812 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:30.443413 17812 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:30.445027 17489 ts_manager.cc:194] Registered new tserver with Master: bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765)
I20260927 13:46:30.446216 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:56731
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:46:30.478035 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:30.782704 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:30.782938 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:30.783098 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:30.790125 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:30.790227 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:30.790381 17817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:46:30.799702 17817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:30.800899 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:30.802101 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:30.814633 17822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:30.814585 17825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:30.814671 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:30.815397 17817 server_base.cc:1034] running on GCE node
I20260927 13:46:30.815963 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:30.816805 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:30.818017 17817 hybrid_clock.cc:648] HybridClock initialized: now 1790516790817985 us; error 48 us; skew 500 ppm
I20260927 13:46:30.818305 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:30 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:30.821254 17817 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:46:30.822566 17817 webserver.cc:460] Webserver started at http://127.1.237.195:38785/ using document root <none> and password file <none>
I20260927 13:46:30.823227 17817 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:30.823352 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:30.823624 17817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:46:30.826225 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "c26305613a444a1d8a507de8d9d2489b"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "3298902d359add6629ecb28f4f2b3631"
server_key_iv: "c559001dd41f4ab916e6fa2e98a14b38"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.826908 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "c26305613a444a1d8a507de8d9d2489b"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "3298902d359add6629ecb28f4f2b3631"
server_key_iv: "c559001dd41f4ab916e6fa2e98a14b38"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.831604 17817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 13:46:30.834735 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.836037 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:30.836194 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c26305613a444a1d8a507de8d9d2489b"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "3298902d359add6629ecb28f4f2b3631"
server_key_iv: "c559001dd41f4ab916e6fa2e98a14b38"
server_key_version: "encryptionkey@0"
I20260927 13:46:30.836347 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:30.849134 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:30.853091 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:30.853312 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:30.854135 17817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:30.855315 17817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:30.855378 17817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.855450 17817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:30.855492 17817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:30.887326 17817 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:36915
I20260927 13:46:30.887387 17944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:36915 every 8 connection(s)
I20260927 13:46:30.888576 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:46:30.896941 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17817
I20260927 13:46:30.897125 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:46:30.897671 1975 external_mini_cluster.cc:1428] Setting key 18b2ba071fb0f74c03c698a565011c1b
Sep 27 13:46:30 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:30.908005 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.890875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:43341 (local address 127.1.237.254:46413)
0927 13:46:30.891045 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.891049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.892196 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 13:46:30.893871 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.893876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.893963 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.894068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:30.895598 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.896336 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.898535 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.899233 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.899328 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.902828 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:30.902833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:30.902836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:30.902865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:30.905339 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.905954 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.905957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.905959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.906070 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.906413 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.906415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.906417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.906655 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:30.906771 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.907598 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.907772 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:46:30.908977 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.891374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:43341)
0927 13:46:30.892046 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:30.892085 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:30.893622 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:30.894294 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:30.894301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:30.894845 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:30.895438 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.895456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.896488 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.896492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.898361 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.898404 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.899443 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.899452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.899712 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.900444 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:30.900463 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:30.902659 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:30.905468 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:30.905474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:30.905485 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:30.905856 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:30.906158 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:30.906161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:30.906163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:30.906318 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:30.906757 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:30.906763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:30.907032 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 13:46:30.907509 (+ 477us) client_negotiation.cc:240] Negotiation successful
0927 13:46:30.907914 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
I20260927 13:46:30.910259 17945 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:30.910507 17945 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:30.911244 17945 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:30.912755 17489 ts_manager.cc:194] Registered new tserver with Master: c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915)
I20260927 13:46:30.913571 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:43341
I20260927 13:46:30.924938 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:46:30.942132 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.929149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.1.237.254:46413)
0927 13:46:30.929346 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.929354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.929791 (+ 437us) server_negotiation.cc:407] Connection header received
0927 13:46:30.929972 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.929977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.930049 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.930148 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:30.931546 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.932249 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.933611 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.934014 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.934137 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.936780 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:30.936786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:30.936791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:30.936828 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:30.939502 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.940139 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.940145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.940148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.940327 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:30.940717 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:30.940730 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:30.940733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:30.941031 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:30.941283 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.941580 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.941828 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:46:30.945647 17489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:46:30.952243 17489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:46:30.969409 17678 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:30.973302 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.962969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.1.237.194:38765)
0927 13:46:30.963615 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.963626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.964604 (+ 978us) server_negotiation.cc:407] Connection header received
0927 13:46:30.964730 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.964741 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.965019 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.965235 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:30.966236 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.967788 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.970686 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.972152 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.972300 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.972510 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.972831 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.972989 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":457,"threads_started":1}
I20260927 13:46:30.973331 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.962871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:38765 (local address 127.0.0.1:38418)
0927 13:46:30.964135 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:30.964188 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:30.964397 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:30.965525 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:30.965534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:30.965565 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:30.966006 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.966023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.967967 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.967984 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.970510 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.970527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.972377 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.972383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.972580 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.972647 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 13:46:30.972776 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 13:46:30.972945 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":51,"thread_start_us":859,"threads_started":1}
I20260927 13:46:30.974766 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.965930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42217 (local address 127.0.0.1:32916)
0927 13:46:30.966475 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:30.966495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:30.966604 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:30.967554 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:30.967558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:30.967579 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:30.968245 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.968261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.970150 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.970155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.972386 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.972400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.974090 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.974095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.974237 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.974278 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 13:46:30.974352 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 13:46:30.974465 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260927 13:46:30.974823 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.963566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.1.237.195:36915)
0927 13:46:30.965583 (+ 2017us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.965592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.966037 (+ 445us) server_negotiation.cc:407] Connection header received
0927 13:46:30.966925 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.966930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.967144 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.967337 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:30.968386 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.969488 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.972164 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.973592 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.973710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.973882 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.974375 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.974520 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775,"thread_start_us":1639,"threads_started":1}
I20260927 13:46:30.975524 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.966153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.1.237.193:42217)
0927 13:46:30.966813 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:30.966821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:30.966846 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:46:30.966931 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:30.966938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:30.967135 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:30.967347 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:30.968464 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.970003 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.972559 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.973901 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.974030 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.974219 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:30.975042 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 13:46:30.975232 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":349,"threads_started":1}
I20260927 13:46:30.975796 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:30.962945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:36915 (local address 127.0.0.1:40570)
0927 13:46:30.965887 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:30.965908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:30.966782 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:30.967601 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:30.967606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:30.967630 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:30.968206 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.968218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.969656 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.969660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.972019 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:30.972031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:30.974012 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:30.974017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:30.974182 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:30.974215 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:46:30.975258 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0927 13:46:30.975380 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2817,"mutex_wait_us":56,"thread_start_us":2584,"threads_started":1}
I20260927 13:46:30.976989 17747 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb05644d824749a8719c39c9058ebb (DEFAULT_TABLE table=test-table [id=2e86b370d18844f0a685d0c0220c4727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:30.978188 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb05644d824749a8719c39c9058ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:30.980394 17613 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb05644d824749a8719c39c9058ebb (DEFAULT_TABLE table=test-table [id=2e86b370d18844f0a685d0c0220c4727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:30.980574 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 82fb05644d824749a8719c39c9058ebb (DEFAULT_TABLE table=test-table [id=2e86b370d18844f0a685d0c0220c4727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:30.981657 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb05644d824749a8719c39c9058ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:30.981689 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82fb05644d824749a8719c39c9058ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:30.992187 17963 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Bootstrap starting.
I20260927 13:46:30.995085 17963 tablet_bootstrap.cc:654] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:30.995230 17965 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Bootstrap starting.
I20260927 13:46:30.996805 17964 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Bootstrap starting.
I20260927 13:46:30.997474 17963 log.cc:824] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:30.998569 17965 tablet_bootstrap.cc:654] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:30.999809 17964 tablet_bootstrap.cc:654] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:30.999818 17965 log.cc:824] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:31.000692 17963 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: No bootstrap required, opened a new log
I20260927 13:46:31.000958 17963 ts_tablet_manager.cc:1397] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:46:31.000936 17964 log.cc:824] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:31.002683 17965 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: No bootstrap required, opened a new log
I20260927 13:46:31.002939 17965 ts_tablet_manager.cc:1397] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:46:31.003890 17964 tablet_bootstrap.cc:492] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: No bootstrap required, opened a new log
I20260927 13:46:31.004133 17964 ts_tablet_manager.cc:1397] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 13:46:31.008936 17963 raft_consensus.cc:348] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.009341 17963 raft_consensus.cc:374] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:31.009462 17963 raft_consensus.cc:729] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26305613a444a1d8a507de8d9d2489b, State: Initialized, Role: FOLLOWER
I20260927 13:46:31.010049 17963 consensus_queue.cc:260] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.011101 17945 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:31.011844 17963 ts_tablet_manager.cc:1428] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 13:46:31.013401 17965 raft_consensus.cc:348] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.013797 17965 raft_consensus.cc:374] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:31.013895 17965 raft_consensus.cc:729] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cd8e3401864d3ab671c92cb460b8cd, State: Initialized, Role: FOLLOWER
I20260927 13:46:31.014323 17964 raft_consensus.cc:348] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.014478 17965 consensus_queue.cc:260] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.014766 17964 raft_consensus.cc:374] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:31.014868 17964 raft_consensus.cc:729] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb654eae544921bea27e059de97db1, State: Initialized, Role: FOLLOWER
I20260927 13:46:31.015654 17964 consensus_queue.cc:260] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.016240 17965 ts_tablet_manager.cc:1428] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260927 13:46:31.017120 17812 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:31.018083 17964 ts_tablet_manager.cc:1428] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 13:46:31.034741 17971 raft_consensus.cc:482] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:31.034925 17971 raft_consensus.cc:504] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.036440 17971 leader_election.cc:288] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915), a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217)
I20260927 13:46:31.043409 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.036797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:55481 (local address 127.1.237.195:36915)
0927 13:46:31.037021 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.037030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.037455 (+ 425us) server_negotiation.cc:407] Connection header received
0927 13:46:31.037795 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.037803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.037876 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.037940 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:31.038712 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.039839 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.041736 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.042483 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.042664 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.042741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.043021 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.043141 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:46:31.043467 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.037698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:34905 (local address 127.1.237.193:42217)
0927 13:46:31.037884 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.037888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.038032 (+ 144us) server_negotiation.cc:407] Connection header received
0927 13:46:31.038191 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.038195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.038259 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.038325 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:31.038970 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.040009 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.041806 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.042563 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.042735 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.042810 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.043090 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.043228 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:46:31.043574 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.036742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:36915 (local address 127.1.237.194:55481)
0927 13:46:31.037349 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:31.037369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:31.037587 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:31.038139 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:31.038144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:31.038168 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:31.038562 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:31.038576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.039951 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.039954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:31.041582 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:31.041594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.042704 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.042711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:31.042866 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.042888 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:46:31.042997 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 13:46:31.043114 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":2,"thread_start_us":299,"threads_started":1}
I20260927 13:46:31.043629 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.037370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42217 (local address 127.1.237.194:34905)
0927 13:46:31.037946 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:31.037961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:31.038067 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:31.038364 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:31.038370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:31.038400 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:31.038819 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:31.038827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.040210 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.040230 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:31.041646 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:31.041659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.042780 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.042786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:31.042942 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.042964 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:46:31.043064 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 13:46:31.043205 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":20,"thread_start_us":273,"threads_started":1}
I20260927 13:46:31.044464 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb05644d824749a8719c39c9058ebb" candidate_uuid: "bbeb654eae544921bea27e059de97db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26305613a444a1d8a507de8d9d2489b" is_pre_election: true
I20260927 13:46:31.044479 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb05644d824749a8719c39c9058ebb" candidate_uuid: "bbeb654eae544921bea27e059de97db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" is_pre_election: true
I20260927 13:46:31.044909 17900 raft_consensus.cc:2454] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb654eae544921bea27e059de97db1 in term 0.
I20260927 13:46:31.044909 17633 raft_consensus.cc:2454] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbeb654eae544921bea27e059de97db1 in term 0.
I20260927 13:46:31.045519 17701 leader_election.cc:302] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cd8e3401864d3ab671c92cb460b8cd, bbeb654eae544921bea27e059de97db1; no voters:
I20260927 13:46:31.045913 17971 raft_consensus.cc:2781] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:46:31.046059 17971 raft_consensus.cc:482] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:46:31.046168 17971 raft_consensus.cc:3037] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:31.048511 17971 raft_consensus.cc:504] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.049103 17971 leader_election.cc:288] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [CANDIDATE]: Term 1 election: Requested vote from peers c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915), a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217)
I20260927 13:46:31.049474 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb05644d824749a8719c39c9058ebb" candidate_uuid: "bbeb654eae544921bea27e059de97db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26305613a444a1d8a507de8d9d2489b"
I20260927 13:46:31.049538 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fb05644d824749a8719c39c9058ebb" candidate_uuid: "bbeb654eae544921bea27e059de97db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
I20260927 13:46:31.049650 17900 raft_consensus.cc:3037] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:31.049785 17633 raft_consensus.cc:3037] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:31.051895 17900 raft_consensus.cc:2454] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb654eae544921bea27e059de97db1 in term 1.
I20260927 13:46:31.052011 17633 raft_consensus.cc:2454] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbeb654eae544921bea27e059de97db1 in term 1.
I20260927 13:46:31.052263 17701 leader_election.cc:302] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbeb654eae544921bea27e059de97db1, c26305613a444a1d8a507de8d9d2489b; no voters:
I20260927 13:46:31.052536 17971 raft_consensus.cc:2781] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:31.052943 17971 raft_consensus.cc:686] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 1 LEADER]: Becoming Leader. State: Replica: bbeb654eae544921bea27e059de97db1, State: Running, Role: LEADER
I20260927 13:46:31.053385 17971 consensus_queue.cc:237] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:31.057727 17488 catalog_manager.cc:5495] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbeb654eae544921bea27e059de97db1 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "bbeb654eae544921bea27e059de97db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } health_report { overall_health: UNKNOWN } } }
I20260927 13:46:31.076224 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.071371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.1.237.194:38765)
0927 13:46:31.071533 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.071537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.071552 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:46:31.071674 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.071677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.071752 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.071864 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:31.072207 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.072925 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.074323 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.074708 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.074807 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.075321 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:31.075763 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:46:31.075879 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.075943 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.076009 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:46:31.083757 17633 raft_consensus.cc:1261] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Refusing update from remote peer bbeb654eae544921bea27e059de97db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:31.083766 17900 raft_consensus.cc:1261] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Refusing update from remote peer bbeb654eae544921bea27e059de97db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:31.084668 17974 consensus_queue.cc:1035] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:31.085001 17971 consensus_queue.cc:1035] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:31.111258 17489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:31.111544 17489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:31.114993 17489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Sending DeleteTablet for 3 replicas of tablet 82fb05644d824749a8719c39c9058ebb
I20260927 13:46:31.116086 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb05644d824749a8719c39c9058ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:31 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:40570
I20260927 13:46:31.116087 17747 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb05644d824749a8719c39c9058ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:31 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:38418
I20260927 13:46:31.116285 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet 82fb05644d824749a8719c39c9058ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:31 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:32916
I20260927 13:46:31.117141 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17550
I20260927 13:46:31.128644 17989 tablet_replica.cc:330] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: stopping tablet replica
I20260927 13:46:31.130029 17989 raft_consensus.cc:2229] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:31.130607 17989 raft_consensus.cc:2258] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:31.133673 17989 ts_tablet_manager.cc:1905] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:31.142802 17989 ts_tablet_manager.cc:1918] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:31.143030 17989 log.cc:1196] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/82fb05644d824749a8719c39c9058ebb
I20260927 13:46:31.143656 17989 ts_tablet_manager.cc:1939] T 82fb05644d824749a8719c39c9058ebb P a5cd8e3401864d3ab671c92cb460b8cd: Deleting consensus metadata
I20260927 13:46:31.145473 17474 catalog_manager.cc:4841] TS a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217): tablet 82fb05644d824749a8719c39c9058ebb (table test-table [id=2e86b370d18844f0a685d0c0220c4727]) successfully deleted
I20260927 13:46:31.146253 17988 tablet_replica.cc:330] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: stopping tablet replica
I20260927 13:46:31.146484 17987 tablet_replica.cc:330] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: stopping tablet replica
I20260927 13:46:31.146850 17988 raft_consensus.cc:2229] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:31.146914 17987 raft_consensus.cc:2229] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:46:31.147456 17988 raft_consensus.cc:2258] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:31.147604 17987 raft_consensus.cc:2258] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:31.149609 17988 ts_tablet_manager.cc:1905] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:46:31 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:46:31.150697 17987 ts_tablet_manager.cc:1905] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:31.156661 17988 ts_tablet_manager.cc:1918] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:31.156863 17988 log.cc:1196] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/82fb05644d824749a8719c39c9058ebb
I20260927 13:46:31.157411 17988 ts_tablet_manager.cc:1939] T 82fb05644d824749a8719c39c9058ebb P c26305613a444a1d8a507de8d9d2489b: Deleting consensus metadata
I20260927 13:46:31.157688 17987 ts_tablet_manager.cc:1918] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:31.157903 17987 log.cc:1196] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/82fb05644d824749a8719c39c9058ebb
I20260927 13:46:31.158468 17987 ts_tablet_manager.cc:1939] T 82fb05644d824749a8719c39c9058ebb P bbeb654eae544921bea27e059de97db1: Deleting consensus metadata
I20260927 13:46:31.159287 17474 catalog_manager.cc:4841] TS c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915): tablet 82fb05644d824749a8719c39c9058ebb (table test-table [id=2e86b370d18844f0a685d0c0220c4727]) successfully deleted
I20260927 13:46:31.159503 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.118341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.1.237.193:42217)
0927 13:46:31.118545 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.118553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.139884 (+ 21331us) server_negotiation.cc:407] Connection header received
0927 13:46:31.140163 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.140169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.140240 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.140304 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:31.142378 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.143555 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.145601 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.146202 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.146303 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.152062 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:31.152075 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:31.152091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:31.152135 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:31.155788 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.156694 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.156702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.156706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.156873 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.157419 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.157426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.157432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.157916 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:31.158178 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.158886 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.159107 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:46:31.160171 17473 catalog_manager.cc:4841] TS bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765): tablet 82fb05644d824749a8719c39c9058ebb (table test-table [id=2e86b370d18844f0a685d0c0220c4727]) successfully deleted
I20260927 13:46:31.161099 17673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:31.300004 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17550
I20260927 13:46:31.324661 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17684
Sep 27 13:46:31 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:46:31.342017 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.325582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.1.237.194:38765)
0927 13:46:31.325748 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.325752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.325767 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:46:31.325823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.325827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.325893 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.325965 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:31.327741 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.328536 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.330374 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.330904 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.330999 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.336221 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:31.336240 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:31.336286 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:31.336339 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:31.339390 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.340035 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.340044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.340047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.340206 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.340557 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.340562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.340565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.340904 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:31.341244 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.341455 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.341663 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:46:31.342864 17807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:31.453301 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17684
I20260927 13:46:31.471622 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17817
Sep 27 13:46:31 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:46:31.485491 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:31.472355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.1.237.195:36915)
0927 13:46:31.472524 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:31.472528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:31.472545 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:46:31.472672 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:31.472676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:31.472738 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:31.472813 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:31.474038 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.474736 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.475992 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:31.476360 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:31.476446 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:31.480679 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:31.480693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:31.480707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:31.480741 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:31.483127 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.483827 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.483833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.483835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.483957 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:31.484248 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:31.484251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:31.484253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:31.484571 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:31.484764 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:31.485003 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 13:46:31.485144 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:46:31.486100 17940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:31.571918 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17817
I20260927 13:46:31.588017 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17457
I20260927 13:46:31.589104 17519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:31.695660 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17457
I20260927 13:46:51.718562 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--webserver_interface=127.1.237.254
--webserver_port=43469
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:46:52.037748 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:52.038013 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:52.038192 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:52.045341 17995 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:46:52.045435 17995 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:46:52.045523 17995 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:46:52.045624 17995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:46:52.045709 17995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:52.045809 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:52.045895 17995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:46:52.045971 17995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:46:52.055645 17995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:52.056887 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:52.058176 17995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:52.080056 18004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:52.080057 18001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:52.080567 18002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:52.080838 17995 server_base.cc:1034] running on GCE node
I20260927 13:46:52.081710 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:52.083266 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:52.084491 17995 hybrid_clock.cc:648] HybridClock initialized: now 1790516812084418 us; error 47 us; skew 500 ppm
I20260927 13:46:52.084854 17995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:52 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:52.094396 17995 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:46:52.096122 17995 webserver.cc:460] Webserver started at http://127.1.237.254:43469/ using document root <none> and password file <none>
I20260927 13:46:52.096912 17995 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:52.097083 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:52.102866 17995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:46:52.105683 18011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:52.106979 17995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:46:52.107208 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "bbf1a8f08b094e99870a01ca425254f5"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "ced75748b00ea41bcaf9b9137ab8011e"
server_key_iv: "d19ecf7d351c42e46a2f21bc743eb375"
server_key_version: "encryptionkey@0"
I20260927 13:46:52.108233 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:52.122397 17995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:52.126492 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:52.126768 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:52.144932 17995 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46413
I20260927 13:46:52.144963 18062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46413 every 8 connection(s)
I20260927 13:46:52.146363 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:46:52.150082 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 17995
I20260927 13:46:52.150856 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:42217
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=34973
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:46:52.153136 18063 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:46:52.156222 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap starting.
I20260927 13:46:52.184310 18063 log.cc:824] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:52.199822 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:46:52.200421 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap complete.
I20260927 13:46:52.209213 18063 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:52.210144 18063 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Initialized, Role: FOLLOWER
I20260927 13:46:52.210875 18063 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:52.211191 18063 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:46:52.211324 18063 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:46:52.211488 18063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 13:46:52.215443 18063 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:52.215958 18063 leader_election.cc:302] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf1a8f08b094e99870a01ca425254f5; no voters:
I20260927 13:46:52.216543 18063 leader_election.cc:288] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 13:46:52.216738 18067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 13:46:52.218384 18063 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:46:52.219463 18067 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 LEADER]: Becoming Leader. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Running, Role: LEADER
I20260927 13:46:52.220052 18067 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:46:52.222813 18068 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:46:52.223702 18068 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:52.223336 18069 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf1a8f08b094e99870a01ca425254f5. Latest consensus state: current_term: 2 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:46:52.224256 18069 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:46:52.236466 18075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:46:52.245134 18075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:46:52.246646 18075 catalog_manager.cc:1259] Loaded cluster ID: b7a511b6ed2f40e7a4411c2be9d8fa34
I20260927 13:46:52.246791 18075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:46:52.250851 18075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:46:52.252774 18075 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Loaded TSK: 0
I20260927 13:46:52.270885 18084 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:46:52.295490 18075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Generated new TSK 1
I20260927 13:46:52.295897 18075 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 13:46:52.512766 18065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:52.513051 18065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:52.513206 18065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:52.521219 18065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:52.521338 18065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:52.521528 18065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:46:52.531680 18065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=34973
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:52.533026 18065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:52.534408 18065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:52.548049 18090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:52.548780 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:52.548813 18091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:52.549302 18065 server_base.cc:1034] running on GCE node
I20260927 13:46:52.550089 18065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:52.551374 18065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:52.552649 18065 hybrid_clock.cc:648] HybridClock initialized: now 1790516812552584 us; error 41 us; skew 500 ppm
I20260927 13:46:52.553036 18065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:52 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:52.557209 18065 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:46:52.559036 18065 webserver.cc:460] Webserver started at http://127.1.237.193:34973/ using document root <none> and password file <none>
I20260927 13:46:52.559927 18065 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:52.560076 18065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:52.567241 18065 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 13:46:52.570611 18100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:52.572223 18065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 13:46:52.572388 18065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "1a0bce340fd5dec472c654f34bbfcdb5"
server_key_iv: "46197b74e9121f962a02fcaf278c171c"
server_key_version: "encryptionkey@0"
I20260927 13:46:52.573666 18065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:52.594540 18065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:52.599874 18065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:52.600148 18065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:52.601245 18065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:52.603085 18065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:52.603236 18065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 13:46:52.603353 18065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:52.603433 18065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:52.642143 18065 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:42217
I20260927 13:46:52.642227 18212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:42217 every 8 connection(s)
I20260927 13:46:52.643667 18065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:46:52.647732 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18065
I20260927 13:46:52.648353 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:38765
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=33723
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:46:52 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:52.668701 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:52.646701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:35811)
0927 13:46:52.647489 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:52.647534 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:52.649348 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:52.650899 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:52.650907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:52.651492 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:52.652040 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:52.652056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:52.654450 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:52.654454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:52.655946 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:52.655964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:52.656966 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:52.656975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:52.657285 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:52.658001 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:52.658035 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:52.660834 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:52.664237 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:52.664245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:52.664261 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:52.664664 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:52.665099 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:52.665104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:52.665108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:52.665298 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:52.666123 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:52.666133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:52.666471 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 13:46:52.666732 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 13:46:52.667356 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":356,"threads_started":1}
I20260927 13:46:52.668701 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:52.646583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:35811 (local address 127.1.237.254:46413)
0927 13:46:52.647267 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:52.647286 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:52.647705 (+ 419us) server_negotiation.cc:407] Connection header received
0927 13:46:52.649906 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:52.649926 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:52.650373 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:52.650794 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:52.652251 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:52.654271 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:52.656102 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:52.656742 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:52.656876 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:52.661111 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:52.661119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:52.661134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:52.661184 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:52.664070 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:52.664811 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:52.664818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:52.664823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:52.664979 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:52.665422 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:52.665426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:52.665428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:52.665985 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:52.666194 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:52.666965 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 13:46:52.667433 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260927 13:46:52.671962 18213 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:52.672338 18213 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:52.673272 18213 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:52.675751 18028 ts_manager.cc:194] Registered new tserver with Master: a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217)
I20260927 13:46:52.677831 18028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:35811
W20260927 13:46:52.976648 18217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:52.976938 18217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:52.977026 18217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:52.984285 18217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:52.984397 18217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:52.984506 18217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:46:52.995121 18217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:38765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=33723
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:52.996950 18217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:52.998440 18217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:53.011775 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:53.011871 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:53.012401 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:53.012849 18217 server_base.cc:1034] running on GCE node
I20260927 13:46:53.013445 18217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:53.014392 18217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:53.015615 18217 hybrid_clock.cc:648] HybridClock initialized: now 1790516813015548 us; error 81 us; skew 500 ppm
I20260927 13:46:53.015928 18217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:53 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:53.019002 18217 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:46:53.020546 18217 webserver.cc:460] Webserver started at http://127.1.237.194:33723/ using document root <none> and password file <none>
I20260927 13:46:53.021237 18217 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:53.021415 18217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:53.026631 18217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 13:46:53.029320 18232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.030609 18217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 13:46:53.030782 18217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "bbeb654eae544921bea27e059de97db1"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "2cc462b1177c76140e7a408a6a35f225"
server_key_iv: "ebba4dd25a240571d383b2163b8130c2"
server_key_version: "encryptionkey@0"
I20260927 13:46:53.031802 18217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:53.046693 18217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:53.050618 18217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:53.050890 18217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:53.051862 18217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:53.053170 18217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:53.053292 18217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.053445 18217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:53.053534 18217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.086616 18217 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:38765
I20260927 13:46:53.086709 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:38765 every 8 connection(s)
I20260927 13:46:53.088191 18217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:46:53.092788 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18217
I20260927 13:46:53.093449 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:36915
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=38785
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 13:46:53 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:53.108434 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.090818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:54165 (local address 127.1.237.254:46413)
0927 13:46:53.090999 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.091004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.092192 (+ 1188us) server_negotiation.cc:407] Connection header received
0927 13:46:53.094248 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.094254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.094347 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.094460 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:53.096016 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.096749 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.098307 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.098722 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.098854 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.102614 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:53.102619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:53.102623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:53.102657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:53.105266 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.106025 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.106029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.106032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.106167 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.106624 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.106635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.106637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.106913 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:53.107073 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.107846 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.108082 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:46:53.109431 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.091348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:54165)
0927 13:46:53.092029 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:53.092066 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:53.093974 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:53.094721 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:53.094730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:53.095279 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:53.095845 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.095863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.096905 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.096912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.098154 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.098166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.098937 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.098945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.099228 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.099946 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:53.099978 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:53.102389 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:53.105422 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:53.105430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:53.105447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:53.105887 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:53.106299 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:53.106305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:53.106309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:53.106500 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:53.107054 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:53.107062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:53.107469 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 13:46:53.107748 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 13:46:53.108286 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":266,"threads_started":1}
I20260927 13:46:53.111088 18345 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:53.111434 18345 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:53.112190 18345 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:53.113718 18028 ts_manager.cc:194] Registered new tserver with Master: bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765)
I20260927 13:46:53.114531 18028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:54165
W20260927 13:46:53.443897 18348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:46:53.444335 18348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:46:53.444541 18348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:46:53.453081 18348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:46:53.453195 18348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:46:53.453310 18348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:46:53.463068 18348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=38785
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:46:53.464448 18348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:46:53.465692 18348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:46:53.478700 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:53.478787 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:46:53.478791 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:46:53.479501 18348 server_base.cc:1034] running on GCE node
I20260927 13:46:53.480124 18348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:46:53.481140 18348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:46:53.482378 18348 hybrid_clock.cc:648] HybridClock initialized: now 1790516813482317 us; error 54 us; skew 500 ppm
I20260927 13:46:53.482718 18348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:46:53 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:46:53.485790 18348 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:46:53.487275 18348 webserver.cc:460] Webserver started at http://127.1.237.195:38785/ using document root <none> and password file <none>
I20260927 13:46:53.487953 18348 fs_manager.cc:359] Metadata directory not provided
I20260927 13:46:53.488109 18348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:46:53.493248 18348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 13:46:53.495882 18363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.497169 18348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:46:53.497354 18348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c26305613a444a1d8a507de8d9d2489b"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "3298902d359add6629ecb28f4f2b3631"
server_key_iv: "c559001dd41f4ab916e6fa2e98a14b38"
server_key_version: "encryptionkey@0"
I20260927 13:46:53.498291 18348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:46:53.528175 18348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:46:53.532076 18348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:46:53.532369 18348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:46:53.533334 18348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:46:53.534660 18348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:46:53.534777 18348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.534914 18348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:46:53.534999 18348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:46:53.568656 18348 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:36915
I20260927 13:46:53.568764 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:36915 every 8 connection(s)
I20260927 13:46:53.570230 18348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:46:53.577061 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18348
Sep 27 13:46:53 dist-test-slave-7g3l krb5kdc[17444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:46:53.593596 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.578221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.1.237.254:46413)
0927 13:46:53.579041 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.579048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.579068 (+ 20us) server_negotiation.cc:407] Connection header received
0927 13:46:53.579118 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.579123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.579212 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.579333 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:53.581094 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.582214 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.583748 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.584316 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.584439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.587444 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:53.587451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:53.587457 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:53.587498 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:53.590484 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.591318 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.591322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.591324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.591436 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.591871 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.591876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.591879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.592274 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:53.592547 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.592904 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.593149 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":582,"threads_started":1}
I20260927 13:46:53.595409 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.573154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:42281 (local address 127.1.237.254:46413)
0927 13:46:53.573498 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.573502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.575102 (+ 1600us) server_negotiation.cc:407] Connection header received
0927 13:46:53.577627 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.577633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.577740 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.577886 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:53.580202 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.581327 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.583340 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.583958 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.584062 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.588246 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:53.588253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:53.588259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:53.588326 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:53.591682 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.592665 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.592671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.592674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.592823 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.593376 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.593382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.593385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.593726 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:53.593868 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.594801 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.595007 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260927 13:46:53.596419 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.573623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:42281)
0927 13:46:53.574838 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:53.574903 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:53.577218 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:53.578436 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:53.578451 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:53.579342 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:46:53.580004 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.580022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.581503 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.581509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.583118 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.583131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.584182 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.584190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.584524 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.585360 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:46:53.585383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:46:53.587991 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:46:53.591866 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:53.591875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:53.591893 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:53.592520 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:53.592940 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:46:53.592945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:46:53.592948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:46:53.593228 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:46:53.593941 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:46:53.593947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:46:53.594379 (+ 432us) client_negotiation.cc:769] Sending connection context
0927 13:46:53.594715 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 13:46:53.595303 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":642,"threads_started":1}
I20260927 13:46:53.597710 18476 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46413
I20260927 13:46:53.597970 18476 heartbeater.cc:464] Registering TS with master...
I20260927 13:46:53.598603 18476 heartbeater.cc:511] Master 127.1.237.254:46413 requested a full tablet report, sending...
I20260927 13:46:53.599965 18028 ts_manager.cc:194] Registered new tserver with Master: c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915)
I20260927 13:46:53.600697 18028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:42281
I20260927 13:46:53.608438 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:46:53.615228 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.609313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.1.237.254:46413)
0927 13:46:53.609494 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.609498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.610101 (+ 603us) server_negotiation.cc:407] Connection header received
0927 13:46:53.610276 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.610280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.610404 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.610522 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:53.610916 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.611844 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.613526 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.613952 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.614064 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.614524 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:53.614716 (+ 192us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 13:46:53.614986 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45490: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 13:46:53.615329 18215 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45490: token expired
W20260927 13:46:53.615871 17948 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.237.254:46413)
I20260927 13:46:53.628682 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.616325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.1.237.254:46413)
0927 13:46:53.616475 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.616481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.616539 (+ 58us) server_negotiation.cc:407] Connection header received
0927 13:46:53.616697 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.616702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.616782 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.616877 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:53.618203 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.618974 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.620359 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.620770 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.620911 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.623484 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:53.623490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:53.623513 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:53.623553 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:53.626001 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.626720 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.626726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.626730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.626852 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:53.627216 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:53.627223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:53.627226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:53.627572 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:53.627830 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.628135 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.628354 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:46:53.631922 18028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:46:53.639302 18028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:46:53.673053 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.651050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:36915 (local address 127.0.0.1:36270)
0927 13:46:53.663639 (+ 12589us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:53.663806 (+ 167us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:53.664027 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:53.665202 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:53.665208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:53.665231 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:53.665701 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.665717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.667742 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.667749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.670473 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.670485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.672007 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.672013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.672251 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.672329 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 13:46:53.672443 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 13:46:53.672644 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12256,"mutex_wait_us":3,"thread_start_us":12497,"threads_started":1}
I20260927 13:46:53.674043 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.651487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.1.237.195:36915)
0927 13:46:53.662921 (+ 11434us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.662930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.664297 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 13:46:53.664436 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.664447 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.664734 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.664925 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:53.665892 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.667559 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.670631 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.671796 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.671926 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.672186 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.673304 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.673528 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11249,"thread_start_us":11075,"threads_started":1}
I20260927 13:46:53.674566 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.651596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:38765 (local address 127.0.0.1:56302)
0927 13:46:53.666320 (+ 14724us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:53.666341 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:53.666465 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:53.667143 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:53.667179 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:53.667202 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:53.667628 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.667643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.669460 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.669465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.671426 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.671440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.673663 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.673669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.673838 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.673884 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 13:46:53.673966 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 13:46:53.674112 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14598,"mutex_wait_us":1,"thread_start_us":14440,"threads_started":1}
I20260927 13:46:53.675287 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.651578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.1.237.194:38765)
0927 13:46:53.664823 (+ 13245us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.664834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.666602 (+ 1768us) server_negotiation.cc:407] Connection header received
0927 13:46:53.666722 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.666729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.666943 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.667120 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:53.667854 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.669303 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.671607 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.673116 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.673288 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.673514 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.674778 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.674959 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13077,"thread_start_us":13264,"threads_started":1}
I20260927 13:46:53.675602 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.665786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:42217 (local address 127.0.0.1:53734)
0927 13:46:53.666680 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:53.666693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:53.666830 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:53.667423 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:53.667427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:53.667454 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:53.668259 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.668273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.670301 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.670306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.672313 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:53.672326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.674777 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.674783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:53.674951 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.674983 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 13:46:53.675078 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 13:46:53.675231 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":21,"thread_start_us":632,"threads_started":1}
I20260927 13:46:53.676789 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:53.665039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.1.237.193:42217)
0927 13:46:53.665733 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:53.665743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:53.666791 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 13:46:53.666987 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:53.666992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:53.667224 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:53.667416 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:53.668458 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.670158 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.672489 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:53.673969 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:53.674118 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:53.674353 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:53.675142 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 13:46:53.675372 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":360,"threads_started":1}
I20260927 13:46:53.678596 18411 tablet_service.cc:1467] Processing CreateTablet for tablet 059193dda4db4e4da371524826304612 (DEFAULT_TABLE table=test-table [id=f745a1870dde4afb9dfdb846745bc713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:53.679097 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 059193dda4db4e4da371524826304612 (DEFAULT_TABLE table=test-table [id=f745a1870dde4afb9dfdb846745bc713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:53.679749 18148 tablet_service.cc:1467] Processing CreateTablet for tablet 059193dda4db4e4da371524826304612 (DEFAULT_TABLE table=test-table [id=f745a1870dde4afb9dfdb846745bc713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:46:53.680154 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059193dda4db4e4da371524826304612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:53.680426 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059193dda4db4e4da371524826304612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:53.680859 18148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059193dda4db4e4da371524826304612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:46:53.681763 18213 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:53.693351 18492 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Bootstrap starting.
I20260927 13:46:53.695309 18493 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Bootstrap starting.
I20260927 13:46:53.697400 18492 tablet_bootstrap.cc:654] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:53.698559 18493 tablet_bootstrap.cc:654] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:53.698629 18492 log.cc:824] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:53.699829 18493 log.cc:824] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:53.700083 18494 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Bootstrap starting.
I20260927 13:46:53.703008 18492 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: No bootstrap required, opened a new log
I20260927 13:46:53.703361 18492 ts_tablet_manager.cc:1397] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260927 13:46:53.706130 18494 tablet_bootstrap.cc:654] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Neither blocks nor log segments found. Creating new log.
I20260927 13:46:53.707118 18494 log.cc:824] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:46:53.709815 18493 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: No bootstrap required, opened a new log
I20260927 13:46:53.710068 18494 tablet_bootstrap.cc:492] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: No bootstrap required, opened a new log
I20260927 13:46:53.710079 18493 ts_tablet_manager.cc:1397] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260927 13:46:53.710311 18494 ts_tablet_manager.cc:1397] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 13:46:53.712988 18492 raft_consensus.cc:348] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.713475 18492 raft_consensus.cc:374] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:53.713603 18492 raft_consensus.cc:729] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cd8e3401864d3ab671c92cb460b8cd, State: Initialized, Role: FOLLOWER
I20260927 13:46:53.714403 18492 consensus_queue.cc:260] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.716965 18492 ts_tablet_manager.cc:1428] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260927 13:46:53.720511 18493 raft_consensus.cc:348] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.720920 18493 raft_consensus.cc:374] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:53.721020 18493 raft_consensus.cc:729] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeb654eae544921bea27e059de97db1, State: Initialized, Role: FOLLOWER
I20260927 13:46:53.721307 18494 raft_consensus.cc:348] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.721732 18494 raft_consensus.cc:374] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:46:53.721609 18493 consensus_queue.cc:260] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.721824 18494 raft_consensus.cc:729] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26305613a444a1d8a507de8d9d2489b, State: Initialized, Role: FOLLOWER
I20260927 13:46:53.722615 18345 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:53.723304 18493 ts_tablet_manager.cc:1428] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Time spent starting tablet: real 0.013s user 0.014s sys 0.002s
I20260927 13:46:53.722386 18494 consensus_queue.cc:260] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:53.725052 18476 heartbeater.cc:503] Master 127.1.237.254:46413 was elected leader, sending a full tablet report...
I20260927 13:46:53.725054 18494 ts_tablet_manager.cc:1428] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260927 13:46:53.822042 18477 tablet.cc:2369] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:46:53.840161 18346 tablet.cc:2369] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:46:53.896085 18214 tablet.cc:2369] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:46:54.013252 18498 raft_consensus.cc:482] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:46:54.013579 18498 raft_consensus.cc:504] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:54.015300 18498 leader_election.cc:288] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915), bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765)
I20260927 13:46:54.024289 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.015609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:34381 (local address 127.1.237.195:36915)
0927 13:46:54.016354 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.016359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.016381 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:46:54.016589 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.016594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.016693 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.016789 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:54.017685 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.018814 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.021873 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.023121 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.023332 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.023400 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.023901 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.024019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620}
I20260927 13:46:54.024289 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.015943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:52507 (local address 127.1.237.194:38765)
0927 13:46:54.016194 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.016202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.016583 (+ 381us) server_negotiation.cc:407] Connection header received
0927 13:46:54.016719 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.016727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.016802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.016883 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:46:54.017693 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.018813 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.021873 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.023139 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.023328 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.023400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.023901 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.024019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:46:54.024547 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.015451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:36915 (local address 127.1.237.193:34381)
0927 13:46:54.016244 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:54.016269 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:54.016420 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:54.016809 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:54.016812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:54.016831 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:54.017525 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:54.017540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.018958 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.018963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:54.021710 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:54.021726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.023426 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.023433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:54.023665 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.023747 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 13:46:54.023848 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 13:46:54.024046 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":288,"spinlock_wait_cycles":33152,"thread_start_us":758,"threads_started":1}
I20260927 13:46:54.024569 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.015540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:38765 (local address 127.1.237.193:52507)
0927 13:46:54.016480 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 13:46:54.016501 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:46:54.016600 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:46:54.016910 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:46:54.016914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:46:54.016950 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:46:54.017536 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:54.017544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.018958 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.018963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:54.021715 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:46:54.021726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.023440 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.023445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:46:54.023650 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.023745 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 13:46:54.023848 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 13:46:54.024046 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":377,"thread_start_us":347,"threads_started":1}
I20260927 13:46:54.025676 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059193dda4db4e4da371524826304612" candidate_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb654eae544921bea27e059de97db1" is_pre_election: true
I20260927 13:46:54.025678 18431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059193dda4db4e4da371524826304612" candidate_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26305613a444a1d8a507de8d9d2489b" is_pre_election: true
I20260927 13:46:54.026160 18431 raft_consensus.cc:2454] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cd8e3401864d3ab671c92cb460b8cd in term 0.
I20260927 13:46:54.026181 18300 raft_consensus.cc:2454] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cd8e3401864d3ab671c92cb460b8cd in term 0.
I20260927 13:46:54.026898 18102 leader_election.cc:302] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a5cd8e3401864d3ab671c92cb460b8cd, bbeb654eae544921bea27e059de97db1, c26305613a444a1d8a507de8d9d2489b; no voters:
I20260927 13:46:54.027437 18498 raft_consensus.cc:2781] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:46:54.027586 18498 raft_consensus.cc:482] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:46:54.027704 18498 raft_consensus.cc:3037] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:54.029765 18498 raft_consensus.cc:504] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:54.030334 18498 leader_election.cc:288] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [CANDIDATE]: Term 1 election: Requested vote from peers c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915), bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765)
I20260927 13:46:54.030872 18431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059193dda4db4e4da371524826304612" candidate_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26305613a444a1d8a507de8d9d2489b"
I20260927 13:46:54.030872 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059193dda4db4e4da371524826304612" candidate_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbeb654eae544921bea27e059de97db1"
I20260927 13:46:54.031086 18431 raft_consensus.cc:3037] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:54.031090 18300 raft_consensus.cc:3037] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:46:54.034230 18431 raft_consensus.cc:2454] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cd8e3401864d3ab671c92cb460b8cd in term 1.
I20260927 13:46:54.034230 18300 raft_consensus.cc:2454] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cd8e3401864d3ab671c92cb460b8cd in term 1.
I20260927 13:46:54.034662 18101 leader_election.cc:302] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a5cd8e3401864d3ab671c92cb460b8cd, bbeb654eae544921bea27e059de97db1, c26305613a444a1d8a507de8d9d2489b; no voters:
I20260927 13:46:54.035050 18498 raft_consensus.cc:2781] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:46:54.035571 18498 raft_consensus.cc:686] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 1 LEADER]: Becoming Leader. State: Replica: a5cd8e3401864d3ab671c92cb460b8cd, State: Running, Role: LEADER
I20260927 13:46:54.036094 18498 consensus_queue.cc:237] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } }
I20260927 13:46:54.040426 18027 catalog_manager.cc:5495] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193). New cstate: current_term: 1 leader_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cd8e3401864d3ab671c92cb460b8cd" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 42217 } health_report { overall_health: HEALTHY } } }
I20260927 13:46:54.114985 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.109255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.1.237.193:42217)
0927 13:46:54.109426 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.109430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.109448 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:46:54.109601 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.109604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.109710 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.109843 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:46:54.110168 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.110884 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.112636 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.113278 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.113421 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.113916 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:46:54.114411 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:46:54.114568 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.114636 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.114719 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:46:54.123458 18431 raft_consensus.cc:1261] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Refusing update from remote peer a5cd8e3401864d3ab671c92cb460b8cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:54.123458 18300 raft_consensus.cc:1261] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Refusing update from remote peer a5cd8e3401864d3ab671c92cb460b8cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:46:54.124598 18503 consensus_queue.cc:1035] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c26305613a444a1d8a507de8d9d2489b" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:54.124984 18498 consensus_queue.cc:1035] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbeb654eae544921bea27e059de97db1" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:46:54.135715 18506 mvcc.cc:205] Tried to move back new op lower bound from 7333956870638436352 to 7333956870295306240. Current Snapshot: MvccSnapshot[applied={T|T < 7333956870638436352}]
I20260927 13:46:54.136344 18508 mvcc.cc:205] Tried to move back new op lower bound from 7333956870638436352 to 7333956870295306240. Current Snapshot: MvccSnapshot[applied={T|T < 7333956870638436352}]
I20260927 13:46:54.142099 18507 mvcc.cc:205] Tried to move back new op lower bound from 7333956870638436352 to 7333956870295306240. Current Snapshot: MvccSnapshot[applied={T|T < 7333956870638436352}]
I20260927 13:46:54.166184 18027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:54.166461 18027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:46:54.169837 18027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Sending DeleteTablet for 3 replicas of tablet 059193dda4db4e4da371524826304612
I20260927 13:46:54.170944 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet 059193dda4db4e4da371524826304612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:54 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:56302
I20260927 13:46:54.171021 18411 tablet_service.cc:1514] Processing DeleteTablet for tablet 059193dda4db4e4da371524826304612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:54 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:36270
I20260927 13:46:54.171895 18516 tablet_replica.cc:330] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: stopping tablet replica
I20260927 13:46:54.172303 18148 tablet_service.cc:1514] Processing DeleteTablet for tablet 059193dda4db4e4da371524826304612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:46:54 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:53734
I20260927 13:46:54.172372 18516 raft_consensus.cc:2229] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:54.172879 18516 raft_consensus.cc:2258] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:54.173877 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18065
I20260927 13:46:54.175493 18516 ts_tablet_manager.cc:1905] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:54.184418 18516 ts_tablet_manager.cc:1918] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:54.184669 18516 log.cc:1196] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/059193dda4db4e4da371524826304612
I20260927 13:46:54.185292 18516 ts_tablet_manager.cc:1939] T 059193dda4db4e4da371524826304612 P bbeb654eae544921bea27e059de97db1: Deleting consensus metadata
I20260927 13:46:54.187461 18012 catalog_manager.cc:4841] TS bbeb654eae544921bea27e059de97db1 (127.1.237.194:38765): tablet 059193dda4db4e4da371524826304612 (table test-table [id=f745a1870dde4afb9dfdb846745bc713]) successfully deleted
I20260927 13:46:54.192494 18518 tablet_replica.cc:330] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: stopping tablet replica
I20260927 13:46:54.192963 18517 tablet_replica.cc:330] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: stopping tablet replica
I20260927 13:46:54.193465 18517 raft_consensus.cc:2229] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:46:54.194039 18517 raft_consensus.cc:2258] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:54.195377 18518 raft_consensus.cc:2229] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:46:54.196122 18518 raft_consensus.cc:2258] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:46:54.196489 18517 ts_tablet_manager.cc:1905] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:54.198139 18518 ts_tablet_manager.cc:1905] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:46:54.203519 18517 ts_tablet_manager.cc:1918] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:54.203737 18517 log.cc:1196] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/059193dda4db4e4da371524826304612
I20260927 13:46:54.204344 18517 ts_tablet_manager.cc:1939] T 059193dda4db4e4da371524826304612 P c26305613a444a1d8a507de8d9d2489b: Deleting consensus metadata
I20260927 13:46:54.206287 18013 catalog_manager.cc:4841] TS c26305613a444a1d8a507de8d9d2489b (127.1.237.195:36915): tablet 059193dda4db4e4da371524826304612 (table test-table [id=f745a1870dde4afb9dfdb846745bc713]) successfully deleted
I20260927 13:46:54.207944 18518 ts_tablet_manager.cc:1918] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:46:54.208128 18518 log.cc:1196] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/059193dda4db4e4da371524826304612
I20260927 13:46:54.208690 18518 ts_tablet_manager.cc:1939] T 059193dda4db4e4da371524826304612 P a5cd8e3401864d3ab671c92cb460b8cd: Deleting consensus metadata
I20260927 13:46:54.210531 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.175014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.1.237.193:42217)
0927 13:46:54.175236 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.175242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.191797 (+ 16555us) server_negotiation.cc:407] Connection header received
0927 13:46:54.192011 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.192018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.192123 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.192258 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:54.194714 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.195808 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.197883 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.198582 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.198693 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.202361 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:54.202374 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:54.202396 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:54.202434 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:54.207348 (+ 4914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.208189 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.208199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.208202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.208380 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.208844 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.208852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.208858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.209282 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:54.209588 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.209918 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.210144 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:46:54.210582 18013 catalog_manager.cc:4841] TS a5cd8e3401864d3ab671c92cb460b8cd (127.1.237.193:42217): tablet 059193dda4db4e4da371524826304612 (table test-table [id=f745a1870dde4afb9dfdb846745bc713]) successfully deleted
I20260927 13:46:54.211409 18208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:54.309803 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18065
I20260927 13:46:54.327939 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18217
I20260927 13:46:54.341331 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.328653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.1.237.194:38765)
0927 13:46:54.328827 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.328831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.328847 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:46:54.329045 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.329049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.329119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.329213 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:54.330567 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.331399 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.332724 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.333116 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.333212 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.336025 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:54.336039 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:54.336056 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:54.336093 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:54.338754 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.339564 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.339572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.339574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.339725 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.340066 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.340069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.340071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.340413 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:54.340625 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.340904 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.341063 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:46:54.342023 18340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:54.429559 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18217
I20260927 13:46:54.446825 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18348
I20260927 13:46:54.459570 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:46:54.447649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.1.237.195:36915)
0927 13:46:54.447822 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:46:54.447826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:46:54.447844 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:46:54.447982 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:46:54.447986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:46:54.448050 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:46:54.448132 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:46:54.449377 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.450083 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.451513 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:46:54.451882 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:46:54.451977 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:46:54.454421 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:46:54.454437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:46:54.454452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:46:54.454496 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:46:54.457105 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.457765 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.457772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.457774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.457907 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:46:54.458252 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:46:54.458256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:46:54.458258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:46:54.458614 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:46:54.458828 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:46:54.459068 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 13:46:54.459309 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:46:54.460209 18471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:54.548128 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18348
I20260927 13:46:54.564065 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 17995
I20260927 13:46:54.565173 18058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:46:54.678592 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 17995
I20260927 13:47:14.700039 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--webserver_interface=127.1.237.254
--webserver_port=43469
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:15.013707 18524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:15.013962 18524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:15.014045 18524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:15.020838 18524 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 13:47:15.020920 18524 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 13:47:15.020957 18524 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 13:47:15.021008 18524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:15.021059 18524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:15.021127 18524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:15.021178 18524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:15.021224 18524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:15.030345 18524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=43469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:15.031417 18524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:15.032554 18524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:15.043952 18529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.043988 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.044078 18530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:15.044410 18524 server_base.cc:1034] running on GCE node
I20260927 13:47:15.045295 18524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:15.046538 18524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:15.047807 18524 hybrid_clock.cc:648] HybridClock initialized: now 1790516835047743 us; error 49 us; skew 500 ppm
I20260927 13:47:15.048149 18524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:15 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:15.051362 18524 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:47:15.052681 18524 webserver.cc:460] Webserver started at http://127.1.237.254:43469/ using document root <none> and password file <none>
I20260927 13:47:15.053329 18524 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:15.053462 18524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:15.058684 18524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:47:15.061390 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:15.062568 18524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:47:15.062745 18524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "bbf1a8f08b094e99870a01ca425254f5"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "ced75748b00ea41bcaf9b9137ab8011e"
server_key_iv: "d19ecf7d351c42e46a2f21bc743eb375"
server_key_version: "encryptionkey@0"
I20260927 13:47:15.063732 18524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:15.090690 18524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:15.094858 18524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:15.095127 18524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:15.112625 18524 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46413
I20260927 13:47:15.112661 18590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46413 every 8 connection(s)
I20260927 13:47:15.113983 18524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:15.120206 18591 sys_catalog.cc:263] Verifying existing consensus state
I20260927 13:47:15.121479 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18524
I20260927 13:47:15.122466 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:42217
--local_ip_for_outbound_sockets=127.1.237.193
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=34973
--webserver_interface=127.1.237.193
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:47:15.124442 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap starting.
I20260927 13:47:15.146239 18591 log.cc:824] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:15.161243 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 13:47:15.161746 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Bootstrap complete.
I20260927 13:47:15.170663 18591 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:47:15.171588 18591 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Initialized, Role: FOLLOWER
I20260927 13:47:15.172217 18591 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:47:15.172433 18591 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:15.172544 18591 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:15.172735 18591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 2 FOLLOWER]: Advancing to term 3
I20260927 13:47:15.176266 18591 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:47:15.176721 18591 leader_election.cc:302] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf1a8f08b094e99870a01ca425254f5; no voters:
I20260927 13:47:15.177366 18591 leader_election.cc:288] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 13:47:15.177496 18595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 13:47:15.178529 18595 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [term 3 LEADER]: Becoming Leader. State: Replica: bbf1a8f08b094e99870a01ca425254f5, State: Running, Role: LEADER
I20260927 13:47:15.179100 18595 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } }
I20260927 13:47:15.179467 18591 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:15.181655 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf1a8f08b094e99870a01ca425254f5. Latest consensus state: current_term: 3 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:47:15.182080 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:15.182710 18596 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bbf1a8f08b094e99870a01ca425254f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf1a8f08b094e99870a01ca425254f5" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46413 } } }
I20260927 13:47:15.183048 18596 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:15.187718 18600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:15.198200 18600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:15.199785 18600 catalog_manager.cc:1259] Loaded cluster ID: b7a511b6ed2f40e7a4411c2be9d8fa34
I20260927 13:47:15.199955 18600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:15.203940 18600 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:15.205543 18600 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Loaded TSK: 0
I20260927 13:47:15.205790 18600 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbf1a8f08b094e99870a01ca425254f5: Loaded TSK: 1
F20260927 13:47:15.207149 18600 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2f4d9de9ad google::LogMessage::Fail() at ??:0
@ 0x7f2f4d9e188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2f4d9de4de google::LogMessage::Flush() at ??:0
@ 0x7f2f4d9dfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2f517427d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2f5173173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2f634111c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2f63410783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2f6349c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260927 13:47:15.251785 18615 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f2f6349c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f2f634e65e9 std::function<>::operator()() at ??:0
@ 0x7f2f63414cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2f63412c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2f6349a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2f6349a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2f5062a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2f5062d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2f5062d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2f50608eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2f538e26db start_thread at ??:0
@ 0x7f2f4c87861f clone at ??:0
W20260927 13:47:15.522540 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:15.522790 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:15.522876 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:15.529820 18593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:15.529909 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:15.530017 18593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:15.539045 18593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=34973
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:15.540161 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:15.541309 18593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:15.553146 18639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.553203 18638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.553200 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:15.556471 18593 server_base.cc:1034] running on GCE node
I20260927 13:47:15.557020 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:15.557910 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:15.559207 18593 hybrid_clock.cc:648] HybridClock initialized: now 1790516835559089 us; error 51 us; skew 500 ppm
I20260927 13:47:15.559500 18593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:15 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:15.562371 18593 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:47:15.563730 18593 webserver.cc:460] Webserver started at http://127.1.237.193:34973/ using document root <none> and password file <none>
I20260927 13:47:15.564612 18593 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:15.564736 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:15.569461 18593 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 13:47:15.571938 18648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:15.573112 18593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:47:15.573275 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "a5cd8e3401864d3ab671c92cb460b8cd"
format_stamp: "Formatted at 2026-09-27 13:46:29 on dist-test-slave-7g3l"
server_key: "1a0bce340fd5dec472c654f34bbfcdb5"
server_key_iv: "46197b74e9121f962a02fcaf278c171c"
server_key_version: "encryptionkey@0"
I20260927 13:47:15.574162 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:15.588809 18593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:15.592267 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:15.592515 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:15.593371 18593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:15.594540 18593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:15.594664 18593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:15.594800 18593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:15.594892 18593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:15.627511 18593 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:42217
I20260927 13:47:15.627611 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:42217 every 8 connection(s)
I20260927 13:47:15.628984 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:15.633492 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:15.631780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:58303)
0927 13:47:15.632467 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:15.632685 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
W20260927 13:47:15.634222 18761 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:46413 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:46413: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
I20260927 13:47:15.634929 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:15.634642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:52471)
0927 13:47:15.634765 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:15.634835 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 13:47:15.635771 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:15.635484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:59209)
0927 13:47:15.635598 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:15.635667 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 13:47:15.635996 18761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:47:15.639003 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18593
I20260927 13:47:15.639689 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:38765
--local_ip_for_outbound_sockets=127.1.237.194
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=33723
--webserver_interface=127.1.237.194
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:15.949285 18764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:15.949522 18764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:15.949700 18764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:15.956561 18764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:15.956656 18764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:15.956831 18764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:15.966260 18764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:38765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=33723
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:15.967403 18764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:15.968631 18764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:15.981417 18769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.981467 18772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:15.981493 18770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:15.982431 18764 server_base.cc:1034] running on GCE node
I20260927 13:47:15.982932 18764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:15.983839 18764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:15.985029 18764 hybrid_clock.cc:648] HybridClock initialized: now 1790516835984985 us; error 42 us; skew 500 ppm
I20260927 13:47:15.985327 18764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:15 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:15.988300 18764 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:47:15.989692 18764 webserver.cc:460] Webserver started at http://127.1.237.194:33723/ using document root <none> and password file <none>
I20260927 13:47:15.990306 18764 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:15.990445 18764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:15.995808 18764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:15.999430 18779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.001119 18764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:47:16.001287 18764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "bbeb654eae544921bea27e059de97db1"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "2cc462b1177c76140e7a408a6a35f225"
server_key_iv: "ebba4dd25a240571d383b2163b8130c2"
server_key_version: "encryptionkey@0"
I20260927 13:47:16.002518 18764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:16.021809 18764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:16.025753 18764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:16.025974 18764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:16.026942 18764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:16.028239 18764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:16.028321 18764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.028465 18764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:16.028559 18764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.060130 18764 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:38765
I20260927 13:47:16.060243 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:38765 every 8 connection(s)
I20260927 13:47:16.061450 18764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:16.066052 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.064334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:59181)
0927 13:47:16.065055 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.065266 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":321,"threads_started":1}
W20260927 13:47:16.066639 18892 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:46413 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:46413: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
I20260927 13:47:16.067384 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.067053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:59983)
0927 13:47:16.067233 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.067292 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:47:16.068066 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.067806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:53569)
0927 13:47:16.067911 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.067975 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 13:47:16.068291 18892 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:47:16.070708 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18764
I20260927 13:47:16.071353 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:36915
--local_ip_for_outbound_sockets=127.1.237.195
--tserver_master_addrs=127.1.237.254:46413
--webserver_port=38785
--webserver_interface=127.1.237.195
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:16.379284 18895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:16.379608 18895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:16.379763 18895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:16.386780 18895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:16.386926 18895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:16.387115 18895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:16.397372 18895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:36915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=38785
--tserver_master_addrs=127.1.237.254:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:16.398591 18895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:16.399901 18895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:16.412755 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:16.412808 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:16.412796 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:16.413018 18895 server_base.cc:1034] running on GCE node
I20260927 13:47:16.413869 18895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:16.414747 18895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:16.415966 18895 hybrid_clock.cc:648] HybridClock initialized: now 1790516836415911 us; error 43 us; skew 500 ppm
I20260927 13:47:16.416289 18895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:16 dist-test-slave-7g3l krb5kdc[17444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:16.419380 18895 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:47:16.420676 18895 webserver.cc:460] Webserver started at http://127.1.237.195:38785/ using document root <none> and password file <none>
I20260927 13:47:16.421327 18895 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:16.421468 18895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:16.426271 18895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:47:16.428875 18910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.430088 18895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:47:16.430255 18895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c26305613a444a1d8a507de8d9d2489b"
format_stamp: "Formatted at 2026-09-27 13:46:30 on dist-test-slave-7g3l"
server_key: "3298902d359add6629ecb28f4f2b3631"
server_key_iv: "c559001dd41f4ab916e6fa2e98a14b38"
server_key_version: "encryptionkey@0"
I20260927 13:47:16.431210 18895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:16.457515 18895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:16.461493 18895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:16.461755 18895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:16.462626 18895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:16.463811 18895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:16.463928 18895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.464053 18895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:16.464138 18895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:16.495729 18895 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:36915
I20260927 13:47:16.495801 19022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:36915 every 8 connection(s)
I20260927 13:47:16.497123 18895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:16.501284 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.499756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:36785)
0927 13:47:16.500354 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.500558 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
W20260927 13:47:16.501875 19023 heartbeater.cc:644] Failed to heartbeat to 127.1.237.254:46413 (0 consecutive failures): Network error: Failed to ping master at 127.1.237.254:46413: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
I20260927 13:47:16.502594 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.502295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:52343)
0927 13:47:16.502430 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.502506 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 13:47:16.502763 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 18895
I20260927 13:47:16.503568 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.503210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:41905)
0927 13:47:16.503332 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.503424 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 13:47:16.503832 19023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 13:47:16.637135 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:16.636444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:33643)
0927 13:47:16.636840 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:16.636935 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":195,"threads_started":1}
I20260927 13:47:17.069515 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:17.068782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:50997)
0927 13:47:17.069213 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:17.069304 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260927 13:47:17.505004 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:17.504273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:44147)
0927 13:47:17.504698 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:17.504777 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":210,"threads_started":1}
I20260927 13:47:17.638592 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:17.637935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:52387)
0927 13:47:17.638352 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:17.638426 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
I20260927 13:47:18.071035 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:18.070400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:59889)
0927 13:47:18.070811 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:18.070882 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260927 13:47:18.506412 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:18.505789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:40637)
0927 13:47:18.506197 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:18.506270 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260927 13:47:18.640110 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:18.639473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:57473)
0927 13:47:18.639855 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:18.639921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":218,"threads_started":1}
I20260927 13:47:19.072465 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:19.071801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:58139)
0927 13:47:19.072214 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:19.072289 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260927 13:47:19.508038 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:19.507370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:57243)
0927 13:47:19.507800 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:19.507874 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":231,"threads_started":1}
I20260927 13:47:19.641546 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:19.640874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:59279)
0927 13:47:19.641290 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:19.641364 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":225,"threads_started":1}
I20260927 13:47:20.073909 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:20.073243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:36453)
0927 13:47:20.073660 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:20.073732 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260927 13:47:20.509500 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:20.508866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:48159)
0927 13:47:20.509255 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:20.509327 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":198,"threads_started":1}
I20260927 13:47:20.643203 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:20.642369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:58329)
0927 13:47:20.642903 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:20.642984 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":289,"threads_started":1}
I20260927 13:47:21.075611 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:21.074849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:51119)
0927 13:47:21.075360 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:21.075438 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":232,"threads_started":1}
I20260927 13:47:21.510986 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:21.510328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:57765)
0927 13:47:21.510750 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:21.510824 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":204,"threads_started":1}
I20260927 13:47:21.644722 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:21.644001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:59055)
0927 13:47:21.644467 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:21.644554 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260927 13:47:22.077112 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:22.076424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:52767)
0927 13:47:22.076812 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:22.076894 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":201,"threads_started":1}
I20260927 13:47:22.512391 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:22.511763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:38863)
0927 13:47:22.512151 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:22.512245 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":203,"threads_started":1}
I20260927 13:47:22.646293 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:22.645537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:48615)
0927 13:47:22.645983 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:22.646100 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":191,"threads_started":1}
I20260927 13:47:23.078742 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:23.077995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:36069)
0927 13:47:23.078465 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:23.078541 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":214,"threads_started":1}
I20260927 13:47:23.513859 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:23.513190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:50341)
0927 13:47:23.513602 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:23.513673 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":179,"threads_started":1}
I20260927 13:47:23.647814 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:23.647099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:48579)
0927 13:47:23.647580 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:23.647658 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
I20260927 13:47:24.080274 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:24.079602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:55477)
0927 13:47:24.080029 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:24.080114 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
I20260927 13:47:24.515326 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:24.514624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:52215)
0927 13:47:24.515029 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:24.515113 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":208,"threads_started":1}
I20260927 13:47:24.649353 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:24.648675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:34137)
0927 13:47:24.649112 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:24.649186 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":212,"threads_started":1}
I20260927 13:47:25.081797 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:25.081096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:43159)
0927 13:47:25.081553 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:25.081631 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":221,"threads_started":1}
I20260927 13:47:25.516805 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:25.516133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:56621)
0927 13:47:25.516569 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:25.516644 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":213,"threads_started":1}
I20260927 13:47:25.650884 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:25.650224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:51613)
0927 13:47:25.650649 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:25.650727 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260927 13:47:26.083500 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:26.082707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:39831)
0927 13:47:26.083136 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:26.083319 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260927 13:47:26.518323 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:26.517663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:34689)
0927 13:47:26.518098 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:26.518175 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":231,"threads_started":1}
I20260927 13:47:26.652462 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:26.651791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:58695)
0927 13:47:26.652219 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:26.652299 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260927 13:47:27.084955 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:27.084317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:59937)
0927 13:47:27.084710 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:27.084793 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260927 13:47:27.519984 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:27.519235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:55069)
0927 13:47:27.519741 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:27.519824 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260927 13:47:27.654001 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:27.653353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:41111)
0927 13:47:27.653766 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:27.653839 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":224,"threads_started":1}
I20260927 13:47:28.086469 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:28.085804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:34737)
0927 13:47:28.086244 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:28.086320 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":260,"threads_started":1}
I20260927 13:47:28.521585 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:28.520796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:52159)
0927 13:47:28.521331 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:28.521409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":303,"threads_started":1}
I20260927 13:47:28.655700 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:28.654867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:54809)
0927 13:47:28.655426 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:28.655514 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260927 13:47:29.088234 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:29.087518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:54607)
0927 13:47:29.087953 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:29.088043 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":218,"threads_started":1}
I20260927 13:47:29.523097 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:29.522398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:47485)
0927 13:47:29.522822 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:29.522913 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260927 13:47:29.657146 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:29.656470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:56817)
0927 13:47:29.656883 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:29.656978 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260927 13:47:30.089727 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:30.089048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:55225)
0927 13:47:30.089480 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:30.089559 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260927 13:47:30.524670 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:30.523957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:46327)
0927 13:47:30.524377 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:30.524466 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":245,"threads_started":1}
I20260927 13:47:30.658846 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:30.658214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.193:55089)
0927 13:47:30.658596 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:30.658675 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":198,"threads_started":1}
I20260927 13:47:31.091455 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:31.090693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.194:34725)
0927 13:47:31.091136 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:31.091281 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":199,"threads_started":1}
I20260927 13:47:31.509406 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18593
I20260927 13:47:31.526768 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:31.525726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46413 (local address 127.1.237.195:55731)
0927 13:47:31.526378 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:31.526512 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.254:46413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":416,"threads_started":1}
I20260927 13:47:31.533459 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:31.510481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.1.237.193:42217)
0927 13:47:31.511203 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:31.511223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:31.511298 (+ 75us) server_negotiation.cc:407] Connection header received
0927 13:47:31.513014 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:31.513032 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:31.513432 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:31.513791 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:31.515306 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.516551 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.517812 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.518502 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.518714 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:31.527821 (+ 9107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:31.527829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:31.527844 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:31.527878 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:31.530529 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.531290 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.531297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.531303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.531470 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.531802 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.531805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.531807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.532331 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:31.532541 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:31.532861 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 13:47:31.533198 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":321,"threads_started":1}
I20260927 13:47:31.534909 18756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:31.616865 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18593
I20260927 13:47:31.634147 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18764
I20260927 13:47:31.651247 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:31.635199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.1.237.194:38765)
0927 13:47:31.635724 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:31.635737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:31.635772 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:47:31.637219 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:31.637228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:31.637642 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:31.638002 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:31.639561 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.640787 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.642057 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.642693 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.642900 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:31.645319 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:31.645326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:31.645341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:31.645374 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:31.648002 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.648735 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.648742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.648748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.648902 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.649285 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.649296 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.649299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.649749 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:31.650085 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:31.650497 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 13:47:31.650872 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260927 13:47:31.652577 18887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:31.733119 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18764
I20260927 13:47:31.749651 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 18895
I20260927 13:47:31.766124 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:31.750668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.1.237.195:36915)
0927 13:47:31.751344 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:31.751357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:31.751397 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:47:31.752963 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:31.752972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:31.753383 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:31.753664 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:31.754867 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.756134 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.757300 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:31.757878 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:31.757991 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:31.760416 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:31.760424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:31.760437 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:31.760477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:31.763475 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.764019 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.764025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.764032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.764178 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:31.764466 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:31.764469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:31.764471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:31.764924 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:31.765129 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:31.765462 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 13:47:31.765847 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":316,"threads_started":1}
I20260927 13:47:31.767474 19018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:31.848784 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 18895
2026-09-27T13:47:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64246 ms)
[----------] 30 tests from SecurityITest (284659 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:31 dist-test-slave-7g3l krb5kdc[19080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:31 dist-test-slave-7g3l krb5kdc[19080](info): set up 2 sockets
Sep 27 13:47:31 dist-test-slave-7g3l krb5kdc[19080](info): commencing operation
krb5kdc: starting...
W20260927 13:47:32.935258 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:33 dist-test-slave-7g3l krb5kdc[19080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:47:33.164336 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:37089
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:37089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:33.498135 19093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:33.498384 19093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:33.498556 19093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:33.505472 19093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:33.505563 19093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:33.505674 19093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:33.505779 19093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:33.505863 19093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:33.515173 19093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:37089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:37089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:33.516355 19093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:33.517637 19093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:33.529013 19101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:33.529088 19098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:33.529063 19099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:33.530251 19093 server_base.cc:1034] running on GCE node
I20260927 13:47:33.530889 19093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:33.532207 19093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:33.533488 19093 hybrid_clock.cc:648] HybridClock initialized: now 1790516853533441 us; error 42 us; skew 500 ppm
I20260927 13:47:33.533766 19093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:33 dist-test-slave-7g3l krb5kdc[19080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:33.536899 19093 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:47:33.538326 19093 webserver.cc:460] Webserver started at http://127.1.237.254:33401/ using document root <none> and password file <none>
I20260927 13:47:33.538995 19093 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:33.539152 19093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:33.539491 19093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:33.542284 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "7afcd43575dc4b53839e2458d0ac2d2a"
format_stamp: "Formatted at 2026-09-27 13:47:33 on dist-test-slave-7g3l"
server_key: "9a42a0d81b4f555ce45e54e35002e58e"
server_key_iv: "8e39515e0f30115c947df4d3d59d3e8e"
server_key_version: "encryptionkey@0"
I20260927 13:47:33.543081 19093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "7afcd43575dc4b53839e2458d0ac2d2a"
format_stamp: "Formatted at 2026-09-27 13:47:33 on dist-test-slave-7g3l"
server_key: "9a42a0d81b4f555ce45e54e35002e58e"
server_key_iv: "8e39515e0f30115c947df4d3d59d3e8e"
server_key_version: "encryptionkey@0"
I20260927 13:47:33.547761 19093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:47:33.550917 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:33.552230 19093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:33.552399 19093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "7afcd43575dc4b53839e2458d0ac2d2a"
format_stamp: "Formatted at 2026-09-27 13:47:33 on dist-test-slave-7g3l"
server_key: "9a42a0d81b4f555ce45e54e35002e58e"
server_key_iv: "8e39515e0f30115c947df4d3d59d3e8e"
server_key_version: "encryptionkey@0"
I20260927 13:47:33.552609 19093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:33.569725 19093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:33.573894 19093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:33.574163 19093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:33.591634 19093 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:37089
I20260927 13:47:33.591655 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:37089 every 8 connection(s)
I20260927 13:47:33.592911 19093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:33.594012 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19093
I20260927 13:47:33.594234 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:33.594856 1975 external_mini_cluster.cc:1428] Setting key b0688af231657f76ce747ec97a28cfa4
I20260927 13:47:33.596836 19160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:47:33 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:33.609539 19160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: Bootstrap starting.
I20260927 13:47:33.613328 19160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:33.614568 19160 log.cc:824] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:33.617625 19160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: No bootstrap required, opened a new log
I20260927 13:47:33.618569 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:33.596744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.1.237.254:37089)
0927 13:47:33.597559 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:33.597572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:33.597617 (+ 45us) server_negotiation.cc:407] Connection header received
0927 13:47:33.599676 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:33.599700 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:33.600198 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:33.600632 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:33.601913 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:33.603928 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:33.605259 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:33.606277 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:33.606471 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:33.610389 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:33.610400 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:33.610418 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:33.610459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:33.614279 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:33.615072 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:33.615083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:33.615094 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:33.615304 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:33.615728 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:33.615731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:33.615733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:33.616328 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:33.616627 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:33.616966 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 13:47:33.617346 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":360,"threads_started":1}
I20260927 13:47:33.625372 19160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } }
I20260927 13:47:33.625653 19160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:33.625722 19160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afcd43575dc4b53839e2458d0ac2d2a, State: Initialized, Role: FOLLOWER
I20260927 13:47:33.626209 19160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } }
I20260927 13:47:33.626405 19160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:33.626509 19160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:33.626662 19160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:33.628851 19160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } }
I20260927 13:47:33.629305 19160 leader_election.cc:302] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7afcd43575dc4b53839e2458d0ac2d2a; no voters:
I20260927 13:47:33.629841 19160 leader_election.cc:288] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:33.629951 19165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:33.630605 19165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [term 1 LEADER]: Becoming Leader. State: Replica: 7afcd43575dc4b53839e2458d0ac2d2a, State: Running, Role: LEADER
I20260927 13:47:33.631127 19165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } }
I20260927 13:47:33.631780 19160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:33.634413 19166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } } }
I20260927 13:47:33.634802 19166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:33.635345 19167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afcd43575dc4b53839e2458d0ac2d2a. Latest consensus state: current_term: 1 leader_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afcd43575dc4b53839e2458d0ac2d2a" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 37089 } } }
I20260927 13:47:33.637346 19173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:33.637374 19167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:33.642382 19173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:33.651368 19173 catalog_manager.cc:1347] Generated new cluster ID: 9a5848ac17ab47f099c6e4477dff62ac
I20260927 13:47:33.651491 19173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:33.666420 19173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:33.667645 19173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:33.683938 19173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: Generated new TSK 0
I20260927 13:47:33.684794 19173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:47:33.752637 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:34.072417 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:34.072710 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:34.072875 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:34.079751 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:34.079886 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:34.080065 19186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:34.089021 19186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:34.090194 19186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:34.091465 19186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:34.103931 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:34.103978 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:34.104238 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:34.105012 19186 server_base.cc:1034] running on GCE node
I20260927 13:47:34.105604 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:34.106496 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:34.107734 19186 hybrid_clock.cc:648] HybridClock initialized: now 1790516854107669 us; error 47 us; skew 500 ppm
I20260927 13:47:34.108054 19186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:34 dist-test-slave-7g3l krb5kdc[19080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:34.111049 19186 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:47:34.112475 19186 webserver.cc:460] Webserver started at http://127.1.237.193:46873/ using document root <none> and password file <none>
I20260927 13:47:34.113118 19186 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:34.113271 19186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:34.113583 19186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:34.116276 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "0822cb0738c34ea8b9743b549b746920"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "9ef98e8bf8a0d48ba8c1cde79b53451a"
server_key_iv: "635fce9ff1136d65c592a642235b28bf"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.116981 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "0822cb0738c34ea8b9743b549b746920"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "9ef98e8bf8a0d48ba8c1cde79b53451a"
server_key_iv: "635fce9ff1136d65c592a642235b28bf"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.121642 19186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:47:34.124802 19201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.126029 19186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:47:34.126217 19186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "0822cb0738c34ea8b9743b549b746920"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "9ef98e8bf8a0d48ba8c1cde79b53451a"
server_key_iv: "635fce9ff1136d65c592a642235b28bf"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.126412 19186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:34.154191 19186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:34.157984 19186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:34.158245 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:34.159090 19186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:34.160341 19186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:34.160442 19186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.160585 19186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:34.160666 19186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.192564 19186 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:45767
I20260927 13:47:34.192644 19313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:45767 every 8 connection(s)
I20260927 13:47:34.193774 19186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:34.197014 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19186
I20260927 13:47:34.197232 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:34.198004 1975 external_mini_cluster.cc:1428] Setting key b4d3a4a1d28afea182ebe7cdb1796f30
Sep 27 13:47:34 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:34.214305 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:34.196183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:40695 (local address 127.1.237.254:37089)
0927 13:47:34.196655 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:34.196659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:34.197598 (+ 939us) server_negotiation.cc:407] Connection header received
0927 13:47:34.199553 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:34.199558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:34.199643 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:34.199760 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:34.201124 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.201816 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.203467 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.203865 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.204015 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:34.207526 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:34.207535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:34.207543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:34.207588 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:34.211123 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:34.211833 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:34.211842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:34.211848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:34.211987 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:34.212384 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:34.212391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:34.212397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:34.212772 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:34.212948 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:34.213584 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 13:47:34.213833 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":232,"threads_started":1}
I20260927 13:47:34.214829 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:34.196644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:37089 (local address 127.1.237.193:40695)
0927 13:47:34.197454 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:34.197488 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:34.199285 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:34.199980 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:34.199988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:34.200446 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:34.200956 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:34.200971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.201988 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.201993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:34.203318 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:34.203330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.204108 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.204117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:34.204359 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:34.205063 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:34.205085 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:34.207319 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:34.211291 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:34.211297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:34.211308 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:34.211701 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:34.212090 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:34.212094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:34.212096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:34.212261 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:34.212906 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:34.212912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:34.213194 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 13:47:34.213406 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 13:47:34.213758 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260927 13:47:34.216406 19314 heartbeater.cc:347] Connected to a master server at 127.1.237.254:37089
I20260927 13:47:34.216728 19314 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:34.217558 19314 heartbeater.cc:511] Master 127.1.237.254:37089 requested a full tablet report, sending...
I20260927 13:47:34.219795 19125 ts_manager.cc:194] Registered new tserver with Master: 0822cb0738c34ea8b9743b549b746920 (127.1.237.193:45767)
I20260927 13:47:34.221218 19125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:40695
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:47:34.250069 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:34.560201 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:34.560462 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:34.560549 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:34.567090 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:34.567224 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:34.567332 19320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:34.576848 19320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:34.578176 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:34.579450 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:34.592299 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:34.592535 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:34.592830 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:34.593367 19320 server_base.cc:1034] running on GCE node
I20260927 13:47:34.593978 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:34.594905 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:34.596136 19320 hybrid_clock.cc:648] HybridClock initialized: now 1790516854596074 us; error 53 us; skew 500 ppm
I20260927 13:47:34.596443 19320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:34 dist-test-slave-7g3l krb5kdc[19080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:34.599565 19320 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:47:34.601095 19320 webserver.cc:460] Webserver started at http://127.1.237.194:45101/ using document root <none> and password file <none>
I20260927 13:47:34.601750 19320 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:34.601892 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:34.602197 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:34.605068 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "eaeb72a4c6c64322a3345bb1b5a3df27"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "560084799679869eda3de1e1dce9df51"
server_key_iv: "63d75c133978f9579ceb4a729f4c58af"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.605842 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "eaeb72a4c6c64322a3345bb1b5a3df27"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "560084799679869eda3de1e1dce9df51"
server_key_iv: "63d75c133978f9579ceb4a729f4c58af"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.610756 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:47:34.614152 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.615471 19320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:47:34.615639 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "eaeb72a4c6c64322a3345bb1b5a3df27"
format_stamp: "Formatted at 2026-09-27 13:47:34 on dist-test-slave-7g3l"
server_key: "560084799679869eda3de1e1dce9df51"
server_key_iv: "63d75c133978f9579ceb4a729f4c58af"
server_key_version: "encryptionkey@0"
I20260927 13:47:34.615832 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:34.631145 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:34.635102 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:34.635470 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:34.636379 19320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:34.637684 19320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:34.637785 19320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.637928 19320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:34.638006 19320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:34.670820 19320 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:44413
I20260927 13:47:34.670881 19447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:44413 every 8 connection(s)
I20260927 13:47:34.672127 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:34.679440 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19320
I20260927 13:47:34.679617 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:34.680063 1975 external_mini_cluster.cc:1428] Setting key 7c2aae53bc53acb4f017cbcbf6c3f57b
Sep 27 13:47:34 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:34.691764 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:34.674636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:43041 (local address 127.1.237.254:37089)
0927 13:47:34.674804 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:34.674808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:34.675876 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 13:47:34.677609 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:34.677614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:34.677698 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:34.677824 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:34.679222 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.680086 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.681689 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.682232 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.682364 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:34.685739 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:34.685745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:34.685751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:34.685789 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:34.688793 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:34.689473 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:34.689477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:34.689481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:34.689608 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:34.690046 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:34.690051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:34.690056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:34.690300 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:34.690487 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:34.691133 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 13:47:34.691375 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:47:34.692631 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:34.675009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:37089 (local address 127.1.237.194:43041)
0927 13:47:34.675727 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:34.675763 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:34.677352 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:34.678047 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:34.678055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:34.678543 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:34.679049 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:34.679066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.680236 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.680242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:34.681545 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:34.681557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:34.682441 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:34.682447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:34.682672 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:34.683304 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:34.683324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:34.685533 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:34.688924 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:34.688934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:34.688953 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:34.689340 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:34.689731 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:34.689736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:34.689740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:34.689921 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:34.690429 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:34.690439 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:34.690747 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 13:47:34.691037 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 13:47:34.691530 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":316,"threads_started":1}
I20260927 13:47:34.694037 19448 heartbeater.cc:347] Connected to a master server at 127.1.237.254:37089
I20260927 13:47:34.694331 19448 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:34.695199 19448 heartbeater.cc:511] Master 127.1.237.254:37089 requested a full tablet report, sending...
I20260927 13:47:34.696758 19125 ts_manager.cc:194] Registered new tserver with Master: eaeb72a4c6c64322a3345bb1b5a3df27 (127.1.237.194:44413)
I20260927 13:47:34.697902 19125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:43041
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:47:34.731930 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:35.043612 19453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:35.043855 19453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:35.044045 19453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:35.051049 19453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:35.051142 19453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:35.051383 19453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:35.060240 19453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:35.061352 19453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:35.062537 19453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:35.075469 19461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:35.075551 19458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:35.075524 19459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:35.076148 19453 server_base.cc:1034] running on GCE node
I20260927 13:47:35.076682 19453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:35.077509 19453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:35.078716 19453 hybrid_clock.cc:648] HybridClock initialized: now 1790516855078668 us; error 34 us; skew 500 ppm
I20260927 13:47:35.079002 19453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:35 dist-test-slave-7g3l krb5kdc[19080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:35.081952 19453 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:47:35.083288 19453 webserver.cc:460] Webserver started at http://127.1.237.195:44561/ using document root <none> and password file <none>
I20260927 13:47:35.083992 19453 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:35.084129 19453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:35.084432 19453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:35.087242 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5"
format_stamp: "Formatted at 2026-09-27 13:47:35 on dist-test-slave-7g3l"
server_key: "c5b67e63a716a0fbf7819525192618b9"
server_key_iv: "822064ca9e4b40f3039602feea7db9b0"
server_key_version: "encryptionkey@0"
I20260927 13:47:35.087996 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5"
format_stamp: "Formatted at 2026-09-27 13:47:35 on dist-test-slave-7g3l"
server_key: "c5b67e63a716a0fbf7819525192618b9"
server_key_iv: "822064ca9e4b40f3039602feea7db9b0"
server_key_version: "encryptionkey@0"
I20260927 13:47:35.092576 19453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:47:35.095818 19468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:35.096989 19453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:35.097182 19453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5"
format_stamp: "Formatted at 2026-09-27 13:47:35 on dist-test-slave-7g3l"
server_key: "c5b67e63a716a0fbf7819525192618b9"
server_key_iv: "822064ca9e4b40f3039602feea7db9b0"
server_key_version: "encryptionkey@0"
I20260927 13:47:35.097368 19453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:35.115410 19453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:35.118891 19453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:35.119256 19453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:35.120142 19453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:35.121373 19453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:35.121471 19453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:35.121590 19453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:35.121662 19453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:35.155493 19453 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:32797
I20260927 13:47:35.155539 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:32797 every 8 connection(s)
I20260927 13:47:35.156880 19453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:35.161798 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19453
I20260927 13:47:35.161988 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:35.162410 1975 external_mini_cluster.cc:1428] Setting key ef9c54498d3c8ad1ddabbf0f330c3293
Sep 27 13:47:35 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:35.175233 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.159224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:58539 (local address 127.1.237.254:37089)
0927 13:47:35.159381 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.159386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.160463 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 13:47:35.162366 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.162372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.162458 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.162565 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:35.164051 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.164865 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.166321 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.166720 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.166850 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.170075 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:35.170080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:35.170083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:35.170124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:35.172627 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.173262 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.173265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.173267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.173380 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.173779 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.173786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.173791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.174047 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:35.174153 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.174793 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.174936 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 13:47:35.176282 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.159676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:37089 (local address 127.1.237.195:58539)
0927 13:47:35.160302 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.160342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.162105 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.162788 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.162796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.163341 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:35.163887 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.163902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.165008 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.165025 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.166176 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.166187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.166935 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.166945 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.167240 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.167801 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:35.167826 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:35.169902 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:35.172747 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:35.172754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:35.172764 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:35.173153 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:35.173479 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:35.173483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:35.173487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:35.173642 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:35.174190 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:35.174195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:35.174477 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.174694 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.175025 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260927 13:47:35.177464 19581 heartbeater.cc:347] Connected to a master server at 127.1.237.254:37089
I20260927 13:47:35.177719 19581 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:35.178354 19581 heartbeater.cc:511] Master 127.1.237.254:37089 requested a full tablet report, sending...
I20260927 13:47:35.179776 19125 ts_manager.cc:194] Registered new tserver with Master: c21cecf418ea4272b6caeb3cd9b0ccb5 (127.1.237.195:32797)
I20260927 13:47:35.180552 19125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:58539
I20260927 13:47:35.189319 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:35.206262 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.193996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.1.237.254:37089)
0927 13:47:35.194159 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.194163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.194669 (+ 506us) server_negotiation.cc:407] Connection header received
0927 13:47:35.194869 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.194879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.194935 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.195036 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:35.196397 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.197059 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.198432 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.198827 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.198965 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.201349 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:35.201355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:35.201359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:35.201394 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:35.203764 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.204408 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.204415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.204418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.204558 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.204922 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.204929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.204933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.205207 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:35.205464 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.205775 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.205979 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:47:35.209656 19125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:35.216235 19125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:35.224290 19314 heartbeater.cc:503] Master 127.1.237.254:37089 was elected leader, sending a full tablet report...
I20260927 13:47:35.245589 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.227420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:44413 (local address 127.0.0.1:52114)
0927 13:47:35.235029 (+ 7609us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.235089 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.235325 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.236503 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.236526 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.236566 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:35.236963 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.236981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.239249 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.239254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.241965 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.241977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.244534 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.244540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.244761 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.244812 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.244962 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.245146 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7312,"thread_start_us":7167,"threads_started":1}
I20260927 13:47:35.246384 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.227621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.1.237.194:44413)
0927 13:47:35.234568 (+ 6947us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.234581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.235548 (+ 967us) server_negotiation.cc:407] Connection header received
0927 13:47:35.235676 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.235687 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.235976 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.236190 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:35.237205 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.239064 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.242136 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.243992 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.244156 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.244393 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.245928 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.246109 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6744,"thread_start_us":6571,"threads_started":1}
I20260927 13:47:35.250205 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.227703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:32797 (local address 127.0.0.1:42050)
0927 13:47:35.238449 (+ 10746us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.238468 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.238575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.242366 (+ 3791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.242375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.242409 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:35.242929 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.242944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.245315 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.245320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.247505 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.247519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.249498 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.249505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.249654 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.249685 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.249772 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.249883 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10623,"thread_start_us":10939,"threads_started":1}
I20260927 13:47:35.250476 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.228346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.1.237.195:32797)
0927 13:47:35.241351 (+ 13005us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.241363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.241403 (+ 40us) server_negotiation.cc:407] Connection header received
0927 13:47:35.241520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.241530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.241799 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.242035 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:35.243400 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.245180 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.247666 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.249001 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.249128 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.249344 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.249983 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.250147 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12784,"thread_start_us":14731,"threads_started":1}
I20260927 13:47:35.250082 19383 tablet_service.cc:1467] Processing CreateTablet for tablet adac1387707c41aba6aa1cd7f00547d1 (DEFAULT_TABLE table=test-table [id=2c10d1167b7948dfbfe19bf2d91a0dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:35.251652 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac1387707c41aba6aa1cd7f00547d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:35.252063 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.239733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42252)
0927 13:47:35.244885 (+ 5152us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.244901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.245000 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.245635 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.245641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.245670 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:35.246098 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.246113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.248038 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.248055 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.249486 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.249499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.251268 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.251274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.251438 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.251472 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.251585 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.251718 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5043,"mutex_wait_us":39,"thread_start_us":4958,"threads_started":1}
I20260927 13:47:35.252099 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.235736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.1.237.193:45767)
0927 13:47:35.244669 (+ 8933us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.244677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.245106 (+ 429us) server_negotiation.cc:407] Connection header received
0927 13:47:35.245196 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.245203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.245425 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.246284 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:35.246489 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.247902 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.249657 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.251056 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.251247 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.251458 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.251628 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.251793 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8770,"thread_start_us":8616,"threads_started":1}
I20260927 13:47:35.254089 19516 tablet_service.cc:1467] Processing CreateTablet for tablet adac1387707c41aba6aa1cd7f00547d1 (DEFAULT_TABLE table=test-table [id=2c10d1167b7948dfbfe19bf2d91a0dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:35.255296 19516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac1387707c41aba6aa1cd7f00547d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:35.255715 19249 tablet_service.cc:1467] Processing CreateTablet for tablet adac1387707c41aba6aa1cd7f00547d1 (DEFAULT_TABLE table=test-table [id=2c10d1167b7948dfbfe19bf2d91a0dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:35.256917 19249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adac1387707c41aba6aa1cd7f00547d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:35.265367 19599 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Bootstrap starting.
I20260927 13:47:35.267848 19600 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Bootstrap starting.
I20260927 13:47:35.269452 19599 tablet_bootstrap.cc:654] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:35.270634 19601 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Bootstrap starting.
I20260927 13:47:35.270756 19599 log.cc:824] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:35.271297 19600 tablet_bootstrap.cc:654] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:35.272352 19600 log.cc:824] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:35.273926 19601 tablet_bootstrap.cc:654] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:35.274125 19599 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: No bootstrap required, opened a new log
I20260927 13:47:35.274379 19599 ts_tablet_manager.cc:1397] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:47:35.274938 19601 log.cc:824] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:35.275242 19600 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: No bootstrap required, opened a new log
I20260927 13:47:35.276031 19600 ts_tablet_manager.cc:1397] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 13:47:35.277277 19601 tablet_bootstrap.cc:492] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: No bootstrap required, opened a new log
I20260927 13:47:35.277521 19601 ts_tablet_manager.cc:1397] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 13:47:35.283897 19601 raft_consensus.cc:348] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.284230 19601 raft_consensus.cc:374] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:35.284348 19601 raft_consensus.cc:729] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0822cb0738c34ea8b9743b549b746920, State: Initialized, Role: FOLLOWER
I20260927 13:47:35.284669 19599 raft_consensus.cc:348] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.285039 19599 raft_consensus.cc:374] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:35.284895 19601 consensus_queue.cc:260] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.285130 19599 raft_consensus.cc:729] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21cecf418ea4272b6caeb3cd9b0ccb5, State: Initialized, Role: FOLLOWER
I20260927 13:47:35.285753 19599 consensus_queue.cc:260] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.286064 19600 raft_consensus.cc:348] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.286541 19600 raft_consensus.cc:374] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:35.286696 19600 raft_consensus.cc:729] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaeb72a4c6c64322a3345bb1b5a3df27, State: Initialized, Role: FOLLOWER
I20260927 13:47:35.287737 19601 ts_tablet_manager.cc:1428] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260927 13:47:35.287577 19600 consensus_queue.cc:260] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.290920 19581 heartbeater.cc:503] Master 127.1.237.254:37089 was elected leader, sending a full tablet report...
I20260927 13:47:35.291271 19448 heartbeater.cc:503] Master 127.1.237.254:37089 was elected leader, sending a full tablet report...
I20260927 13:47:35.291451 19599 ts_tablet_manager.cc:1428] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260927 13:47:35.291944 19600 ts_tablet_manager.cc:1428] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260927 13:47:35.408702 19582 tablet.cc:2369] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:35.424125 19449 tablet.cc:2369] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:35.446769 19315 tablet.cc:2369] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:35.481684 19605 raft_consensus.cc:482] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:35.482007 19605 raft_consensus.cc:504] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.483739 19605 leader_election.cc:288] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaeb72a4c6c64322a3345bb1b5a3df27 (127.1.237.194:44413), c21cecf418ea4272b6caeb3cd9b0ccb5 (127.1.237.195:32797)
I20260927 13:47:35.492936 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.484331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:56469 (local address 127.1.237.195:32797)
0927 13:47:35.484517 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.484525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.485299 (+ 774us) server_negotiation.cc:407] Connection header received
0927 13:47:35.485495 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.485500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.485577 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.485676 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:35.486563 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.487671 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.490499 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.491799 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.491965 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.492041 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.492544 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.492674 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:47:35.492936 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.484215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:60859 (local address 127.1.237.194:44413)
0927 13:47:35.484416 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.484422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.485294 (+ 872us) server_negotiation.cc:407] Connection header received
0927 13:47:35.485484 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.485489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.485577 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.485676 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:35.486495 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.487636 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.490483 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.491794 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.491974 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.492041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.492544 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.492674 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:47:35.493096 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.484174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:32797 (local address 127.1.237.193:56469)
0927 13:47:35.485182 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.485198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.485310 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.485706 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.485710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.485749 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:35.486388 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.486402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.487795 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.487800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.490310 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.490323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.492071 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.492077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.492327 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.492380 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.492479 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.492620 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":102,"spinlock_wait_cycles":3840,"thread_start_us":444,"threads_started":1}
I20260927 13:47:35.493378 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.484058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:44413 (local address 127.1.237.193:60859)
0927 13:47:35.485120 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.485152 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:35.485348 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:35.485706 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:35.485710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:35.485743 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:35.486289 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.486306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.487796 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.487800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.490263 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:35.490280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.492060 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.492067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:35.492342 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.492392 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 13:47:35.492479 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:47:35.492620 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":317,"spinlock_wait_cycles":36992,"thread_start_us":730,"threads_started":1}
I20260927 13:47:35.494270 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac1387707c41aba6aa1cd7f00547d1" candidate_uuid: "0822cb0738c34ea8b9743b549b746920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" is_pre_election: true
I20260927 13:47:35.494273 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac1387707c41aba6aa1cd7f00547d1" candidate_uuid: "0822cb0738c34ea8b9743b549b746920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" is_pre_election: true
I20260927 13:47:35.494746 19536 raft_consensus.cc:2454] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0822cb0738c34ea8b9743b549b746920 in term 0.
I20260927 13:47:35.494756 19403 raft_consensus.cc:2454] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0822cb0738c34ea8b9743b549b746920 in term 0.
I20260927 13:47:35.495474 19203 leader_election.cc:302] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0822cb0738c34ea8b9743b549b746920, c21cecf418ea4272b6caeb3cd9b0ccb5; no voters:
I20260927 13:47:35.495848 19605 raft_consensus.cc:2781] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:35.496042 19605 raft_consensus.cc:482] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:35.496155 19605 raft_consensus.cc:3037] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:35.498262 19605 raft_consensus.cc:504] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.498802 19605 leader_election.cc:288] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [CANDIDATE]: Term 1 election: Requested vote from peers eaeb72a4c6c64322a3345bb1b5a3df27 (127.1.237.194:44413), c21cecf418ea4272b6caeb3cd9b0ccb5 (127.1.237.195:32797)
I20260927 13:47:35.499298 19536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac1387707c41aba6aa1cd7f00547d1" candidate_uuid: "0822cb0738c34ea8b9743b549b746920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5"
I20260927 13:47:35.499370 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adac1387707c41aba6aa1cd7f00547d1" candidate_uuid: "0822cb0738c34ea8b9743b549b746920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27"
I20260927 13:47:35.499526 19536 raft_consensus.cc:3037] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:35.499599 19403 raft_consensus.cc:3037] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:35.501966 19536 raft_consensus.cc:2454] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0822cb0738c34ea8b9743b549b746920 in term 1.
I20260927 13:47:35.502048 19403 raft_consensus.cc:2454] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0822cb0738c34ea8b9743b549b746920 in term 1.
I20260927 13:47:35.502410 19203 leader_election.cc:302] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0822cb0738c34ea8b9743b549b746920, c21cecf418ea4272b6caeb3cd9b0ccb5; no voters:
I20260927 13:47:35.502702 19605 raft_consensus.cc:2781] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:35.503202 19605 raft_consensus.cc:686] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 1 LEADER]: Becoming Leader. State: Replica: 0822cb0738c34ea8b9743b549b746920, State: Running, Role: LEADER
I20260927 13:47:35.503672 19605 consensus_queue.cc:237] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } }
I20260927 13:47:35.508111 19125 catalog_manager.cc:5495] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0822cb0738c34ea8b9743b549b746920 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "0822cb0738c34ea8b9743b549b746920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0822cb0738c34ea8b9743b549b746920" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 45767 } health_report { overall_health: HEALTHY } } }
I20260927 13:47:35.539220 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.533594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.1.237.193:45767)
0927 13:47:35.533767 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.533771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.533786 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:47:35.533946 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.533950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.534016 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.534158 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:35.534532 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.535402 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.536914 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.537465 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.537578 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.537986 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:35.538605 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:35.538705 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.538809 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.538865 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:47:35.546819 19403 raft_consensus.cc:1261] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 1 FOLLOWER]: Refusing update from remote peer 0822cb0738c34ea8b9743b549b746920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:35.546823 19536 raft_consensus.cc:1261] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 1 FOLLOWER]: Refusing update from remote peer 0822cb0738c34ea8b9743b549b746920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:35.547753 19610 consensus_queue.cc:1035] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c21cecf418ea4272b6caeb3cd9b0ccb5" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:35.548108 19605 consensus_queue.cc:1035] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaeb72a4c6c64322a3345bb1b5a3df27" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 44413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:35.557655 19613 mvcc.cc:205] Tried to move back new op lower bound from 7333957040311160832 to 7333957040146157568. Current Snapshot: MvccSnapshot[applied={T|T < 7333957040311160832}]
I20260927 13:47:35.572772 19125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:35.573020 19125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:35.576189 19125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7afcd43575dc4b53839e2458d0ac2d2a: Sending DeleteTablet for 3 replicas of tablet adac1387707c41aba6aa1cd7f00547d1
I20260927 13:47:35.577210 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet adac1387707c41aba6aa1cd7f00547d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:35 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:52114
I20260927 13:47:35.577250 19516 tablet_service.cc:1514] Processing DeleteTablet for tablet adac1387707c41aba6aa1cd7f00547d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:35 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:42050
I20260927 13:47:35.577531 19249 tablet_service.cc:1514] Processing DeleteTablet for tablet adac1387707c41aba6aa1cd7f00547d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:35 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:42252
I20260927 13:47:35.579473 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19186
I20260927 13:47:35.589536 19624 tablet_replica.cc:330] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: stopping tablet replica
I20260927 13:47:35.590020 19624 raft_consensus.cc:2229] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:35.590346 19623 tablet_replica.cc:330] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: stopping tablet replica
I20260927 13:47:35.590535 19624 raft_consensus.cc:2258] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:35.590894 19623 raft_consensus.cc:2229] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:35.591457 19623 raft_consensus.cc:2258] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:35.602394 19623 ts_tablet_manager.cc:1905] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:35.604305 19625 tablet_replica.cc:330] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: stopping tablet replica
I20260927 13:47:35.604487 19624 ts_tablet_manager.cc:1905] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:47:35 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:47:35.607923 19625 raft_consensus.cc:2229] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:35.608623 19625 raft_consensus.cc:2258] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:35.610767 19625 ts_tablet_manager.cc:1905] T adac1387707c41aba6aa1cd7f00547d1 P 0822cb0738c34ea8b9743b549b746920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:35.613986 19623 ts_tablet_manager.cc:1918] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:35.616216 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.580608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.1.237.193:45767)
0927 13:47:35.580806 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.580814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.595710 (+ 14896us) server_negotiation.cc:407] Connection header received
0927 13:47:35.595978 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.595985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.596062 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.596131 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:35.597805 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.599004 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.602359 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.603049 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.603135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.608628 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:35.608640 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:35.608654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:35.608705 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:35.612832 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.613689 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.613699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.613704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.613902 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.614343 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.614348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.614351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.614796 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:35.615088 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.615672 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.615885 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:47:35.617549 19309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:35.618795 19623 log.cc:1196] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/adac1387707c41aba6aa1cd7f00547d1
I20260927 13:47:35.618769 19624 ts_tablet_manager.cc:1918] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:35.619024 19624 log.cc:1196] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/adac1387707c41aba6aa1cd7f00547d1
I20260927 13:47:35.619450 19623 ts_tablet_manager.cc:1939] T adac1387707c41aba6aa1cd7f00547d1 P c21cecf418ea4272b6caeb3cd9b0ccb5: Deleting consensus metadata
I20260927 13:47:35.620380 19624 ts_tablet_manager.cc:1939] T adac1387707c41aba6aa1cd7f00547d1 P eaeb72a4c6c64322a3345bb1b5a3df27: Deleting consensus metadata
I20260927 13:47:35.621315 19110 catalog_manager.cc:4841] TS c21cecf418ea4272b6caeb3cd9b0ccb5 (127.1.237.195:32797): tablet adac1387707c41aba6aa1cd7f00547d1 (table test-table [id=2c10d1167b7948dfbfe19bf2d91a0dd4]) successfully deleted
I20260927 13:47:35.623139 19109 catalog_manager.cc:4841] TS eaeb72a4c6c64322a3345bb1b5a3df27 (127.1.237.194:44413): tablet adac1387707c41aba6aa1cd7f00547d1 (table test-table [id=2c10d1167b7948dfbfe19bf2d91a0dd4]) successfully deleted
I20260927 13:47:35.727887 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19186
W20260927 13:47:35.746902 19109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:45767 (error 108)
I20260927 13:47:35.747193 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19320
I20260927 13:47:35.748871 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.748298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42296)
0927 13:47:35.748575 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.748691 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 13:47:35.749123 19109 catalog_manager.cc:4584] TS 0822cb0738c34ea8b9743b549b746920 (127.1.237.193:45767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adac1387707c41aba6aa1cd7f00547d1: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Sep 27 13:47:35 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:47:35.762015 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.748213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.1.237.194:44413)
0927 13:47:35.748419 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.748426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.748450 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:47:35.748635 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.748642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.748734 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.748832 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:35.750153 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.750878 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.752275 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.752646 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.752739 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.756758 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:35.756778 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:35.756798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:35.756837 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:35.759589 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.760234 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.760240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.760242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.760358 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.760682 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.760685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.760687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.761056 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:35.761270 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.761498 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.761652 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 13:47:35.762722 19443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:35.802935 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.802400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42306)
0927 13:47:35.802635 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.802749 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 13:47:35.804201 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.803767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42322)
0927 13:47:35.803930 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.804061 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 13:47:35.852066 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19320
I20260927 13:47:35.864662 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.864213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42336)
0927 13:47:35.864420 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.864499 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 13:47:35.865691 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.865362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42344)
0927 13:47:35.865510 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.865585 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 13:47:35.868361 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19453
Sep 27 13:47:35 dist-test-slave-7g3l krb5kdc[19080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:47:35.882484 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.869070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.1.237.195:32797)
0927 13:47:35.869222 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:35.869226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:35.869243 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:47:35.869351 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:35.869355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:35.869416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:35.869496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:35.870737 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.871598 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.872947 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:35.873338 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:35.873430 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:35.877646 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:35.877661 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:35.877677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:35.877707 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:35.880201 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.880789 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.880794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.880796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.880923 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:35.881232 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:35.881234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:35.881236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:35.881550 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:35.881755 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:35.881982 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 13:47:35.882135 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 13:47:35.883075 19576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:35.946461 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.945872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42350)
0927 13:47:35.946138 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.946253 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 13:47:35.947713 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:35.947288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:45767 (local address 127.0.0.1:42366)
0927 13:47:35.947496 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:35.947582 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 13:47:35.971312 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19453
I20260927 13:47:35.987720 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19093
I20260927 13:47:35.988966 19155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:36.100016 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19093
2026-09-27T13:47:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T13:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:36Z Disabled control of system clock
I20260927 13:47:36.172120 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36829
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:36.480691 19635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:36.480955 19635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:36.481033 19635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:36.487941 19635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:36.488045 19635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:36.488088 19635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:36.488137 19635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:36.488180 19635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:36.496820 19635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:36.497924 19635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:36.499676 19635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:36.510660 19643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:36.510630 19641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:36.510699 19640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:36.511445 19635 server_base.cc:1034] running on GCE node
I20260927 13:47:36.512091 19635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:36.513288 19635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:36.514519 19635 hybrid_clock.cc:648] HybridClock initialized: now 1790516856514469 us; error 60 us; skew 500 ppm
I20260927 13:47:36.514854 19635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:36.516989 19635 webserver.cc:460] Webserver started at http://127.1.237.254:40257/ using document root <none> and password file <none>
I20260927 13:47:36.517644 19635 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:36.517799 19635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:36.518108 19635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:36.520941 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "3bc4ce25784b45e7bd1feacfe774bb21"
format_stamp: "Formatted at 2026-09-27 13:47:36 on dist-test-slave-7g3l"
server_key: "2b93f90613ccf925461a833ad2d3a993"
server_key_iv: "d1e8cec507acd871bbeb2ee9b0232f86"
server_key_version: "encryptionkey@0"
I20260927 13:47:36.521742 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "3bc4ce25784b45e7bd1feacfe774bb21"
format_stamp: "Formatted at 2026-09-27 13:47:36 on dist-test-slave-7g3l"
server_key: "2b93f90613ccf925461a833ad2d3a993"
server_key_iv: "d1e8cec507acd871bbeb2ee9b0232f86"
server_key_version: "encryptionkey@0"
I20260927 13:47:36.526435 19635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:47:36.529712 19649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:36.530989 19635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:47:36.531200 19635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "3bc4ce25784b45e7bd1feacfe774bb21"
format_stamp: "Formatted at 2026-09-27 13:47:36 on dist-test-slave-7g3l"
server_key: "2b93f90613ccf925461a833ad2d3a993"
server_key_iv: "d1e8cec507acd871bbeb2ee9b0232f86"
server_key_version: "encryptionkey@0"
I20260927 13:47:36.531409 19635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:36.544893 19635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:36.545605 19635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:36.545883 19635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:36.563706 19635 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36829
I20260927 13:47:36.563732 19700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36829 every 8 connection(s)
I20260927 13:47:36.564922 19635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:36.568284 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:36.569751 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19635
I20260927 13:47:36.569938 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:36.570353 1975 external_mini_cluster.cc:1428] Setting key 01b9d32c39e6d30f6c30a910f8f983b9
I20260927 13:47:36.582069 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: Bootstrap starting.
I20260927 13:47:36.584616 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:36.572184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.1.237.254:36829)
0927 13:47:36.573230 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:36.573248 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:36.573295 (+ 47us) server_negotiation.cc:407] Connection header received
0927 13:47:36.574931 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:36.574961 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:36.575573 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:36.576003 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:36.576466 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:36.578248 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:36.579629 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:36.580795 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:36.581002 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:36.581985 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:36.581995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:36.582010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:36.582672 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:36.582761 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:36.582785 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:36.582886 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:36.583095 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 13:47:36.583399 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":321,"threads_started":1}
I20260927 13:47:36.585681 19701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:36.586679 19701 log.cc:824] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:36.589143 19701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: No bootstrap required, opened a new log
I20260927 13:47:36.595923 19701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } }
I20260927 13:47:36.596261 19701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:36.596367 19701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc4ce25784b45e7bd1feacfe774bb21, State: Initialized, Role: FOLLOWER
I20260927 13:47:36.596850 19701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } }
I20260927 13:47:36.597107 19701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:36.597249 19701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:36.597414 19701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:36.599588 19701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } }
I20260927 13:47:36.600040 19701 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc4ce25784b45e7bd1feacfe774bb21; no voters:
I20260927 13:47:36.600538 19701 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:36.600658 19706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:36.601361 19706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc4ce25784b45e7bd1feacfe774bb21, State: Running, Role: LEADER
I20260927 13:47:36.601819 19706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } }
I20260927 13:47:36.602562 19701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:36.604971 19707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } } }
I20260927 13:47:36.604996 19708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc4ce25784b45e7bd1feacfe774bb21. Latest consensus state: current_term: 1 leader_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc4ce25784b45e7bd1feacfe774bb21" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36829 } } }
I20260927 13:47:36.605427 19707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:36.605424 19708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:36.607337 19713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:36.612072 19713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:36.620223 19713 catalog_manager.cc:1347] Generated new cluster ID: 4fe7c87ca4824a379e2600d9dc2cff1a
I20260927 13:47:36.620345 19713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:36.632831 19713 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:36.633955 19713 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:36.646312 19713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: Generated new TSK 0
I20260927 13:47:36.647010 19713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:36.657487 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--builtin_ntp_servers=127.1.237.212:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:36.982327 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:36.982579 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:36.982662 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:36.989610 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:36.989694 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:36.989804 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:36.998291 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:36.999396 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:37.001016 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:37.014047 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.014111 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.014047 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:37.014695 19725 server_base.cc:1034] running on GCE node
I20260927 13:47:37.015265 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:37.016088 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:37.017292 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790516857017254 us; error 30 us; skew 500 ppm
I20260927 13:47:37.017547 19725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:37.019559 19725 webserver.cc:460] Webserver started at http://127.1.237.193:39809/ using document root <none> and password file <none>
I20260927 13:47:37.020144 19725 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:37.020222 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:37.020476 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:37.023096 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "cbeda537bb3259af421ee150fbf80796"
server_key_iv: "29550a8b988def14e424cc490df63fcd"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.023885 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "cbeda537bb3259af421ee150fbf80796"
server_key_iv: "29550a8b988def14e424cc490df63fcd"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.028452 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:47:37.031716 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.033160 19725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:47:37.033327 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "cbeda537bb3259af421ee150fbf80796"
server_key_iv: "29550a8b988def14e424cc490df63fcd"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.033535 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:37.063274 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:37.064071 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:37.064374 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:37.065337 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:37.066756 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:37.066903 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.067072 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:37.067168 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.101364 19725 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:35517
I20260927 13:47:37.101454 19851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:35517 every 8 connection(s)
I20260927 13:47:37.102689 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:37.103818 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19725
I20260927 13:47:37.103986 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:37.104537 1975 external_mini_cluster.cc:1428] Setting key e1c78f1d911873856834cb7ad1d22dbc
I20260927 13:47:37.107532 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--builtin_ntp_servers=127.1.237.212:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:37.117449 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.105791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:51685 (local address 127.1.237.254:36829)
0927 13:47:37.106585 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:37.106591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:37.108007 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 13:47:37.109873 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:37.109881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:37.109985 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:37.110102 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:37.111143 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.112352 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.114152 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.114745 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.114838 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.116523 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:37.116532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:37.116540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:37.116563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:37.116613 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:37.116631 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:37.116725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:37.116923 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 13:47:37.117035 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":557,"threads_started":1}
I20260927 13:47:37.118149 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.106599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36829 (local address 127.1.237.193:51685)
0927 13:47:37.107847 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:37.107893 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:37.109604 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:37.110299 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:37.110307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:37.110392 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:37.110973 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.110993 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.112505 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.112509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.113981 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.113995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.114966 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.115000 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.115341 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.116233 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:37.116236 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:37.116271 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:37.116340 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:37.116728 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:37.116736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:37.116741 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:47:37.116877 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 13:47:37.117136 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":367,"threads_started":1}
I20260927 13:47:37.119560 19852 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36829
I20260927 13:47:37.119833 19852 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:37.120487 19852 heartbeater.cc:511] Master 127.1.237.254:36829 requested a full tablet report, sending...
I20260927 13:47:37.122263 19666 ts_manager.cc:194] Registered new tserver with Master: 54c5aab45d5745a4aa1fbf2bb56b79ec (127.1.237.193:35517)
I20260927 13:47:37.123724 19666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.193:51685
W20260927 13:47:37.421615 19856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:37.421855 19856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:37.422005 19856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:37.428900 19856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:37.428988 19856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:37.429157 19856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:37.438864 19856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:37.440374 19856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:37.442673 19856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:37.455219 19861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.455291 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.455322 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:37.456058 19856 server_base.cc:1034] running on GCE node
I20260927 13:47:37.456655 19856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:37.457494 19856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:37.458685 19856 hybrid_clock.cc:648] HybridClock initialized: now 1790516857458648 us; error 30 us; skew 500 ppm
I20260927 13:47:37.458981 19856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:37.460974 19856 webserver.cc:460] Webserver started at http://127.1.237.194:33863/ using document root <none> and password file <none>
I20260927 13:47:37.461629 19856 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:37.461745 19856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:37.462090 19856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:37.464732 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "cefbe54c910d44bbb8dd8dc78ae8e838"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "47802944761f3b8a5cfcc732bbf14365"
server_key_iv: "15afc32c2d9cb1c087c5e7b25b7a4828"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.465447 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "cefbe54c910d44bbb8dd8dc78ae8e838"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "47802944761f3b8a5cfcc732bbf14365"
server_key_iv: "15afc32c2d9cb1c087c5e7b25b7a4828"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.469978 19856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:47:37.473105 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.474237 19856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 13:47:37.474395 19856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "cefbe54c910d44bbb8dd8dc78ae8e838"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "47802944761f3b8a5cfcc732bbf14365"
server_key_iv: "15afc32c2d9cb1c087c5e7b25b7a4828"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.474629 19856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:37.491757 19856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:37.492465 19856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:37.492753 19856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:37.493587 19856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:37.494910 19856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:37.495028 19856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.495158 19856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:37.495303 19856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 13:47:37.526933 19856 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:35671
I20260927 13:47:37.526983 19982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:35671 every 8 connection(s)
I20260927 13:47:37.528264 19856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:37.530313 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19856
I20260927 13:47:37.530489 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:37.531185 1975 external_mini_cluster.cc:1428] Setting key 6daa036e5c3511a076d6ed1891db694f
I20260927 13:47:37.533706 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--builtin_ntp_servers=127.1.237.212:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:37.540886 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.530981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:53035 (local address 127.1.237.254:36829)
0927 13:47:37.531324 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:37.531331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:37.532730 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 13:47:37.534504 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:37.534509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:37.534591 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:37.534710 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:37.535788 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.536526 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.538145 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.538544 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.538678 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.540087 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:37.540093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:37.540098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:37.540117 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:37.540177 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:37.540194 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:37.540272 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:37.540494 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 13:47:37.540587 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260927 13:47:37.541781 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.531587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36829 (local address 127.1.237.194:53035)
0927 13:47:37.532584 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:37.532631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:37.534263 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:37.534916 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:37.534924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:37.535025 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:37.535635 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.535651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.536693 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.536698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.537998 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.538014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.538753 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.538759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.538975 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.539711 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:37.539713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:37.539748 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:37.539836 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:37.540287 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:37.540298 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:37.540310 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 13:47:37.540473 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:47:37.540820 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":333,"threads_started":1}
I20260927 13:47:37.543098 19983 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36829
I20260927 13:47:37.543381 19983 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:37.544098 19983 heartbeater.cc:511] Master 127.1.237.254:36829 requested a full tablet report, sending...
I20260927 13:47:37.545804 19666 ts_manager.cc:194] Registered new tserver with Master: cefbe54c910d44bbb8dd8dc78ae8e838 (127.1.237.194:35671)
I20260927 13:47:37.547015 19666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.194:53035
W20260927 13:47:37.851449 19986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:37.851694 19986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:37.851776 19986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:37.858649 19986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:37.858731 19986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:37.858839 19986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:37.867688 19986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:36829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:37.868798 19986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:37.870390 19986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:37.883189 19992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.883230 19991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:37.883126 19994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:37.883733 19986 server_base.cc:1034] running on GCE node
I20260927 13:47:37.884370 19986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:37.885300 19986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:37.886530 19986 hybrid_clock.cc:648] HybridClock initialized: now 1790516857886465 us; error 53 us; skew 500 ppm
I20260927 13:47:37.886817 19986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:37.889192 19986 webserver.cc:460] Webserver started at http://127.1.237.195:46615/ using document root <none> and password file <none>
I20260927 13:47:37.889818 19986 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:37.889940 19986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:37.890244 19986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:37.892933 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "4298f0fbdc1e496fa48cde8e100dbd23"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "ba8daf77cba4d390e7234ad87b89191b"
server_key_iv: "5a8b8193a6ba293dd447d61bf4544f4d"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.893693 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "4298f0fbdc1e496fa48cde8e100dbd23"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "ba8daf77cba4d390e7234ad87b89191b"
server_key_iv: "5a8b8193a6ba293dd447d61bf4544f4d"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.898309 19986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:47:37.901394 20000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.902592 19986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:47:37.902750 19986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "4298f0fbdc1e496fa48cde8e100dbd23"
format_stamp: "Formatted at 2026-09-27 13:47:37 on dist-test-slave-7g3l"
server_key: "ba8daf77cba4d390e7234ad87b89191b"
server_key_iv: "5a8b8193a6ba293dd447d61bf4544f4d"
server_key_version: "encryptionkey@0"
I20260927 13:47:37.902949 19986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:37.920624 19986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:37.921327 19986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:37.921620 19986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:37.922487 19986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:37.923830 19986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:37.923949 19986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.924081 19986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:37.924175 19986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:37.955925 19986 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:35831
I20260927 13:47:37.956005 20112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:35831 every 8 connection(s)
I20260927 13:47:37.957144 19986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:37.967396 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 19986
I20260927 13:47:37.967576 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:37.968081 1975 external_mini_cluster.cc:1428] Setting key 90a7855de18ef9bacd0960f251a33331
I20260927 13:47:37.968689 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.959494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:59293 (local address 127.1.237.254:36829)
0927 13:47:37.959660 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:37.959664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:37.960938 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 13:47:37.962595 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:37.962599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:37.962665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:37.962750 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:37.963902 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.964618 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.966025 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.966403 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.966487 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.967902 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:37.967908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:37.967912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:37.967934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:37.967971 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:37.967986 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:37.968117 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:37.968315 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 13:47:37.968407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:47:37.969693 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.959976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:36829 (local address 127.1.237.195:59293)
0927 13:47:37.960786 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:37.960820 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:37.962370 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:37.962957 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:37.962964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:37.963058 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:37.963753 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.963771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.964749 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.964753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.965901 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:37.965913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.966574 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.966578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:37.966755 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.967549 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:37.967553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:37.967608 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:37.967724 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:37.968089 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:37.968097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:37.968102 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:47:37.968287 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 13:47:37.968581 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":247,"threads_started":1}
I20260927 13:47:37.971055 20113 heartbeater.cc:347] Connected to a master server at 127.1.237.254:36829
I20260927 13:47:37.971367 20113 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:37.971967 20113 heartbeater.cc:511] Master 127.1.237.254:36829 requested a full tablet report, sending...
I20260927 13:47:37.973208 19666 ts_manager.cc:194] Registered new tserver with Master: 4298f0fbdc1e496fa48cde8e100dbd23 (127.1.237.195:35831)
I20260927 13:47:37.973933 19666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.195:59293
I20260927 13:47:37.982982 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:37.995975 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:37.990597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.1.237.254:36829)
0927 13:47:37.990750 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:37.990754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:37.991310 (+ 556us) server_negotiation.cc:407] Connection header received
0927 13:47:37.991440 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:37.991444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:37.991502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:37.991582 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:37.992014 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.992658 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.993937 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:37.994337 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:37.994418 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:37.995445 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:37.995449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:37.995452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:37.995467 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:37.995494 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:37.995506 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:37.995574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:37.995698 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 13:47:37.995762 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:47:37.999387 19666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:38.006153 19666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:38.036243 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.017723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:35831 (local address 127.0.0.1:50734)
0927 13:47:38.029219 (+ 11496us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:38.029240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:38.029373 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:38.029987 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:38.029992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:38.030022 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:38.030308 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.030318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.031899 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.031907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.033330 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.033340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.035094 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.035309 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.035429 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 13:47:38.035567 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 13:47:38.035842 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11365,"mutex_wait_us":1,"thread_start_us":8820,"threads_started":1}
I20260927 13:47:38.036403 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.017356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35517 (local address 127.0.0.1:46184)
0927 13:47:38.027460 (+ 10104us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:38.027475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:38.027590 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:38.028633 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:38.028644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:38.028686 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:38.029206 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.029217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.031105 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.031111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.033712 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.033726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.035632 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.035823 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.035853 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:47:38.035940 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 13:47:38.036064 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9990,"mutex_wait_us":90,"thread_start_us":9893,"threads_started":1}
I20260927 13:47:38.036839 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.017343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:35671 (local address 127.0.0.1:55786)
0927 13:47:38.026863 (+ 9520us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:38.026896 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:38.027063 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:38.028848 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:38.028853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:38.028888 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:38.029412 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.029426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.031250 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.031255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.033716 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.033726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.035983 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.036165 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.036200 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:47:38.036290 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 13:47:38.036438 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9275,"mutex_wait_us":178,"thread_start_us":11549,"threads_started":1}
I20260927 13:47:38.037207 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.018629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.1.237.195:35831)
0927 13:47:38.026487 (+ 7858us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.026498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.029361 (+ 2863us) server_negotiation.cc:407] Connection header received
0927 13:47:38.029547 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.029554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.029801 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.029983 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:38.030499 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.031758 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.033521 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.034889 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035050 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.035368 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.036742 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.036858 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7645,"thread_start_us":7480,"threads_started":1}
I20260927 13:47:38.037314 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.018110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.1.237.193:35517)
0927 13:47:38.027979 (+ 9869us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.027987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.028025 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:47:38.028136 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.028145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.028373 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.028573 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:38.029459 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.030911 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.033942 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.035431 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.035794 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.036023 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.036184 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9714,"thread_start_us":9571,"threads_started":1}
I20260927 13:47:38.037727 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.017410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.1.237.194:35671)
0927 13:47:38.028096 (+ 10686us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.028103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.028126 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:47:38.028206 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.028211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.028373 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.028573 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:38.029583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.031101 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.033943 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.035479 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.035605 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.035827 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.037189 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.037361 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10530,"thread_start_us":10582,"threads_started":1}
I20260927 13:47:38.041225 19787 tablet_service.cc:1467] Processing CreateTablet for tablet cceaab8e747d477fbf2404a8a7e5183f (DEFAULT_TABLE table=test-table [id=e40f67149fc64b6b98485a911eba9842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:38.041225 20048 tablet_service.cc:1467] Processing CreateTablet for tablet cceaab8e747d477fbf2404a8a7e5183f (DEFAULT_TABLE table=test-table [id=e40f67149fc64b6b98485a911eba9842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:38.041432 19918 tablet_service.cc:1467] Processing CreateTablet for tablet cceaab8e747d477fbf2404a8a7e5183f (DEFAULT_TABLE table=test-table [id=e40f67149fc64b6b98485a911eba9842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:38.042666 19918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceaab8e747d477fbf2404a8a7e5183f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:38.042662 19787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceaab8e747d477fbf2404a8a7e5183f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:38.042659 20048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cceaab8e747d477fbf2404a8a7e5183f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:38.053853 20131 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Bootstrap starting.
I20260927 13:47:38.057358 20131 tablet_bootstrap.cc:654] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:38.058357 20131 log.cc:824] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:38.059759 20132 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Bootstrap starting.
I20260927 13:47:38.060941 20133 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Bootstrap starting.
I20260927 13:47:38.061543 20131 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: No bootstrap required, opened a new log
I20260927 13:47:38.061794 20131 ts_tablet_manager.cc:1397] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 13:47:38.064328 20133 tablet_bootstrap.cc:654] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:38.064858 20132 tablet_bootstrap.cc:654] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:38.065595 20133 log.cc:824] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:38.066066 20132 log.cc:824] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:38.068358 20133 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: No bootstrap required, opened a new log
I20260927 13:47:38.068627 20133 ts_tablet_manager.cc:1397] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:47:38.069139 20132 tablet_bootstrap.cc:492] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: No bootstrap required, opened a new log
I20260927 13:47:38.069402 20132 ts_tablet_manager.cc:1397] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 13:47:38.071033 20131 raft_consensus.cc:348] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.071483 20131 raft_consensus.cc:374] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:38.071591 20131 raft_consensus.cc:729] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefbe54c910d44bbb8dd8dc78ae8e838, State: Initialized, Role: FOLLOWER
I20260927 13:47:38.072178 20131 consensus_queue.cc:260] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.073246 19983 heartbeater.cc:503] Master 127.1.237.254:36829 was elected leader, sending a full tablet report...
I20260927 13:47:38.073881 20131 ts_tablet_manager.cc:1428] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 13:47:38.076925 20133 raft_consensus.cc:348] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.077286 20133 raft_consensus.cc:374] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:38.077365 20133 raft_consensus.cc:729] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4298f0fbdc1e496fa48cde8e100dbd23, State: Initialized, Role: FOLLOWER
I20260927 13:47:38.077875 20133 consensus_queue.cc:260] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.078927 20113 heartbeater.cc:503] Master 127.1.237.254:36829 was elected leader, sending a full tablet report...
I20260927 13:47:38.079561 20133 ts_tablet_manager.cc:1428] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 13:47:38.079412 20132 raft_consensus.cc:348] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.079793 20132 raft_consensus.cc:374] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:38.079886 20132 raft_consensus.cc:729] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c5aab45d5745a4aa1fbf2bb56b79ec, State: Initialized, Role: FOLLOWER
I20260927 13:47:38.080420 20132 consensus_queue.cc:260] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.081396 19852 heartbeater.cc:503] Master 127.1.237.254:36829 was elected leader, sending a full tablet report...
I20260927 13:47:38.081945 20132 ts_tablet_manager.cc:1428] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
W20260927 13:47:38.104887 19853 tablet.cc:2369] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:38.208787 20114 tablet.cc:2369] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:38.280159 19984 tablet.cc:2369] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:38.298171 20137 raft_consensus.cc:482] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:38.298491 20137 raft_consensus.cc:504] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.300194 20137 leader_election.cc:288] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54c5aab45d5745a4aa1fbf2bb56b79ec (127.1.237.193:35517), 4298f0fbdc1e496fa48cde8e100dbd23 (127.1.237.195:35831)
I20260927 13:47:38.309167 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.300588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:37913 (local address 127.1.237.193:35517)
0927 13:47:38.300802 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.300809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.301448 (+ 639us) server_negotiation.cc:407] Connection header received
0927 13:47:38.301654 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.301659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.301766 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.301864 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:38.302511 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.303572 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.306717 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.307964 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.308115 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.308180 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.308789 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.308914 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 13:47:38.309165 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.301086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:36675 (local address 127.1.237.195:35831)
0927 13:47:38.301310 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.301318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.301734 (+ 416us) server_negotiation.cc:407] Connection header received
0927 13:47:38.301832 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.301839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.301922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.302006 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:38.302651 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.303677 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.306701 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.307964 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.308119 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.308179 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.308789 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.308914 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 13:47:38.309309 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.300627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:35831 (local address 127.1.237.194:36675)
0927 13:47:38.301630 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:38.301644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:38.301742 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:38.302030 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:38.302036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:38.302068 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:38.302492 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.302503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.303848 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.303854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.306483 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.306497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.308192 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.308196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.308456 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.308584 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 13:47:38.308719 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 13:47:38.308934 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":294,"thread_start_us":671,"threads_started":1}
I20260927 13:47:38.309818 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.300535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:35517 (local address 127.1.237.194:37913)
0927 13:47:38.301310 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:38.301333 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:38.301501 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:38.301894 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:38.301897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:38.301926 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:38.302361 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.302378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.303713 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.303717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.306484 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:38.306497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.308186 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.308193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:38.308453 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.308586 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 13:47:38.308719 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:47:38.308934 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":321,"thread_start_us":443,"threads_started":1}
I20260927 13:47:38.310264 20068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceaab8e747d477fbf2404a8a7e5183f" candidate_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" is_pre_election: true
I20260927 13:47:38.310864 20068 raft_consensus.cc:2454] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cefbe54c910d44bbb8dd8dc78ae8e838 in term 0.
I20260927 13:47:38.310811 19807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceaab8e747d477fbf2404a8a7e5183f" candidate_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" is_pre_election: true
I20260927 13:47:38.311420 19807 raft_consensus.cc:2454] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cefbe54c910d44bbb8dd8dc78ae8e838 in term 0.
I20260927 13:47:38.311645 19874 leader_election.cc:302] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4298f0fbdc1e496fa48cde8e100dbd23, cefbe54c910d44bbb8dd8dc78ae8e838; no voters:
I20260927 13:47:38.312026 20137 raft_consensus.cc:2781] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:38.312180 20137 raft_consensus.cc:482] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:38.312307 20137 raft_consensus.cc:3037] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:38.314378 20137 raft_consensus.cc:504] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.314972 20137 leader_election.cc:288] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [CANDIDATE]: Term 1 election: Requested vote from peers 54c5aab45d5745a4aa1fbf2bb56b79ec (127.1.237.193:35517), 4298f0fbdc1e496fa48cde8e100dbd23 (127.1.237.195:35831)
I20260927 13:47:38.315375 19807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceaab8e747d477fbf2404a8a7e5183f" candidate_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec"
I20260927 13:47:38.315418 20068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cceaab8e747d477fbf2404a8a7e5183f" candidate_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4298f0fbdc1e496fa48cde8e100dbd23"
I20260927 13:47:38.315586 20068 raft_consensus.cc:3037] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:38.315586 19807 raft_consensus.cc:3037] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:38.318586 19807 raft_consensus.cc:2454] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cefbe54c910d44bbb8dd8dc78ae8e838 in term 1.
I20260927 13:47:38.318586 20068 raft_consensus.cc:2454] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cefbe54c910d44bbb8dd8dc78ae8e838 in term 1.
I20260927 13:47:38.318964 19871 leader_election.cc:302] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4298f0fbdc1e496fa48cde8e100dbd23, 54c5aab45d5745a4aa1fbf2bb56b79ec, cefbe54c910d44bbb8dd8dc78ae8e838; no voters:
I20260927 13:47:38.319288 20137 raft_consensus.cc:2781] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:38.319737 20137 raft_consensus.cc:686] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 1 LEADER]: Becoming Leader. State: Replica: cefbe54c910d44bbb8dd8dc78ae8e838, State: Running, Role: LEADER
I20260927 13:47:38.320227 20137 consensus_queue.cc:237] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } }
I20260927 13:47:38.324297 19665 catalog_manager.cc:5495] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 reported cstate change: term changed from 0 to 1, leader changed from <none> to cefbe54c910d44bbb8dd8dc78ae8e838 (127.1.237.194). New cstate: current_term: 1 leader_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cefbe54c910d44bbb8dd8dc78ae8e838" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260927 13:47:38.399348 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.394107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.1.237.194:35671)
0927 13:47:38.394341 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.394348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.394460 (+ 112us) server_negotiation.cc:407] Connection header received
0927 13:47:38.394647 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.394651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.394733 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.394853 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:38.395226 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.395944 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.397359 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.397766 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.397855 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.398269 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:38.398758 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:38.398891 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.398969 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.399028 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:47:38.407624 20068 raft_consensus.cc:1261] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 1 FOLLOWER]: Refusing update from remote peer cefbe54c910d44bbb8dd8dc78ae8e838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:38.407712 19807 raft_consensus.cc:1261] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 1 FOLLOWER]: Refusing update from remote peer cefbe54c910d44bbb8dd8dc78ae8e838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:38.408705 20142 consensus_queue.cc:1035] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4298f0fbdc1e496fa48cde8e100dbd23" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:38.409037 20137 consensus_queue.cc:1035] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c5aab45d5745a4aa1fbf2bb56b79ec" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:38.418926 20145 mvcc.cc:205] Tried to move back new op lower bound from 7333957052027187200 to 7333957051683364864. Current Snapshot: MvccSnapshot[applied={T|T < 7333957052027187200}]
I20260927 13:47:38.419471 20146 mvcc.cc:205] Tried to move back new op lower bound from 7333957052027187200 to 7333957051683364864. Current Snapshot: MvccSnapshot[applied={T|T < 7333957052027187200}]
I20260927 13:47:38.437690 19666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:38.437991 19666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:38.441504 19666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bc4ce25784b45e7bd1feacfe774bb21: Sending DeleteTablet for 3 replicas of tablet cceaab8e747d477fbf2404a8a7e5183f
I20260927 13:47:38.442842 19787 tablet_service.cc:1514] Processing DeleteTablet for tablet cceaab8e747d477fbf2404a8a7e5183f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:38 UTC) from {username='slave'} at 127.0.0.1:46184
I20260927 13:47:38.442842 20048 tablet_service.cc:1514] Processing DeleteTablet for tablet cceaab8e747d477fbf2404a8a7e5183f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:38 UTC) from {username='slave'} at 127.0.0.1:50734
I20260927 13:47:38.443921 19918 tablet_service.cc:1514] Processing DeleteTablet for tablet cceaab8e747d477fbf2404a8a7e5183f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:38 UTC) from {username='slave'} at 127.0.0.1:55786
I20260927 13:47:38.444130 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19725
I20260927 13:47:38.454519 20157 tablet_replica.cc:330] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: stopping tablet replica
I20260927 13:47:38.455050 20157 raft_consensus.cc:2229] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:38.455509 20156 tablet_replica.cc:330] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: stopping tablet replica
I20260927 13:47:38.455915 20157 raft_consensus.cc:2258] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:38.455996 20156 raft_consensus.cc:2229] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:38.456529 20156 raft_consensus.cc:2258] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:38.457387 20155 tablet_replica.cc:330] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: stopping tablet replica
I20260927 13:47:38.457769 20155 raft_consensus.cc:2229] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:38.458256 20155 raft_consensus.cc:2258] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:38.458537 20157 ts_tablet_manager.cc:1905] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:38.458689 20156 ts_tablet_manager.cc:1905] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:38.460266 20155 ts_tablet_manager.cc:1905] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:38.465832 20157 ts_tablet_manager.cc:1918] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:38.466046 20157 log.cc:1196] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/cceaab8e747d477fbf2404a8a7e5183f
I20260927 13:47:38.466634 20157 ts_tablet_manager.cc:1939] T cceaab8e747d477fbf2404a8a7e5183f P cefbe54c910d44bbb8dd8dc78ae8e838: Deleting consensus metadata
I20260927 13:47:38.468115 20155 ts_tablet_manager.cc:1918] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:38.468261 20155 log.cc:1196] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/cceaab8e747d477fbf2404a8a7e5183f
I20260927 13:47:38.468452 19652 catalog_manager.cc:4841] TS cefbe54c910d44bbb8dd8dc78ae8e838 (127.1.237.194:35671): tablet cceaab8e747d477fbf2404a8a7e5183f (table test-table [id=e40f67149fc64b6b98485a911eba9842]) successfully deleted
I20260927 13:47:38.468812 20155 ts_tablet_manager.cc:1939] T cceaab8e747d477fbf2404a8a7e5183f P 54c5aab45d5745a4aa1fbf2bb56b79ec: Deleting consensus metadata
I20260927 13:47:38.470552 19650 catalog_manager.cc:4841] TS 54c5aab45d5745a4aa1fbf2bb56b79ec (127.1.237.193:35517): tablet cceaab8e747d477fbf2404a8a7e5183f (table test-table [id=e40f67149fc64b6b98485a911eba9842]) successfully deleted
I20260927 13:47:38.473201 20156 ts_tablet_manager.cc:1918] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:38.473366 20156 log.cc:1196] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/cceaab8e747d477fbf2404a8a7e5183f
I20260927 13:47:38.473762 20156 ts_tablet_manager.cc:1939] T cceaab8e747d477fbf2404a8a7e5183f P 4298f0fbdc1e496fa48cde8e100dbd23: Deleting consensus metadata
I20260927 13:47:38.474943 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.445550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.1.237.193:35517)
0927 13:47:38.445767 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.445776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.467251 (+ 21475us) server_negotiation.cc:407] Connection header received
0927 13:47:38.467642 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.467647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.467745 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.467836 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:38.468572 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.469727 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.471411 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.472040 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.472165 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.473384 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:38.473395 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:38.473410 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:38.474100 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:38.474157 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:38.474187 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:38.474297 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.474482 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.474590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:38.475250 19653 catalog_manager.cc:4841] TS 4298f0fbdc1e496fa48cde8e100dbd23 (127.1.237.195:35831): tablet cceaab8e747d477fbf2404a8a7e5183f (table test-table [id=e40f67149fc64b6b98485a911eba9842]) successfully deleted
I20260927 13:47:38.475994 19847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:38.598694 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19725
I20260927 13:47:38.616255 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19856
I20260927 13:47:38.623049 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.617030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.1.237.194:35671)
0927 13:47:38.617184 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.617189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.617256 (+ 67us) server_negotiation.cc:407] Connection header received
0927 13:47:38.617435 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.617441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.617524 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.617639 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:38.618010 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.618696 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.619999 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.620434 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.620594 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.621664 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:38.621673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:38.621687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:38.622319 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:38.622371 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:38.622390 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:38.622524 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.622660 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.622754 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:47:38.623894 19978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:38.732484 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19856
I20260927 13:47:38.750131 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19986
I20260927 13:47:38.756240 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:38.750823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.1.237.195:35831)
0927 13:47:38.751024 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:38.751029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:38.751047 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:47:38.751146 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:38.751200 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:38.751299 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:38.751384 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:38.751753 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.752447 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.753586 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:38.753947 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:38.754035 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:38.754899 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:38.754907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:38.754921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:38.755549 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:38.755593 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:38.755613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:38.755693 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:38.755813 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 13:47:38.755889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 13:47:38.756959 20108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:38.844427 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19986
I20260927 13:47:38.860463 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 19635
I20260927 13:47:38.861574 19696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:38.970348 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 19635
2026-09-27T13:47:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2869 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:39 dist-test-slave-7g3l krb5kdc[20165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:39 dist-test-slave-7g3l krb5kdc[20165](info): set up 2 sockets
Sep 27 13:47:39 dist-test-slave-7g3l krb5kdc[20165](info): commencing operation
krb5kdc: starting...
W20260927 13:47:40.055244 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:40 dist-test-slave-7g3l krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:47:40.270255 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34863
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:40.576508 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:40.576758 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:40.576908 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:40.583550 20178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:40.583648 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:40.583746 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:40.583863 20178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:40.583918 20178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:40.593022 20178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:40.594163 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:40.595403 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:40.606237 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:40.606199 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:40.606169 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:40.606832 20178 server_base.cc:1034] running on GCE node
I20260927 13:47:40.607592 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:40.608839 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:40.610080 20178 hybrid_clock.cc:648] HybridClock initialized: now 1790516860610044 us; error 48 us; skew 500 ppm
I20260927 13:47:40.610376 20178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:40 dist-test-slave-7g3l krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:40.613276 20178 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:47:40.614665 20178 webserver.cc:460] Webserver started at http://127.1.237.254:46805/ using document root <none> and password file <none>
I20260927 13:47:40.615315 20178 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:40.615411 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:40.615640 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:40.618305 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "176dbb86bc6448a8b959aa576c60a6e8"
format_stamp: "Formatted at 2026-09-27 13:47:40 on dist-test-slave-7g3l"
server_key: "9d36c98c13dae3b58274ae09ee75be80"
server_key_iv: "819da8ffe01665b0966397d325ab14bd"
server_key_version: "encryptionkey@0"
I20260927 13:47:40.618964 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "176dbb86bc6448a8b959aa576c60a6e8"
format_stamp: "Formatted at 2026-09-27 13:47:40 on dist-test-slave-7g3l"
server_key: "9d36c98c13dae3b58274ae09ee75be80"
server_key_iv: "819da8ffe01665b0966397d325ab14bd"
server_key_version: "encryptionkey@0"
I20260927 13:47:40.623592 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:47:40.626750 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:40.628036 20178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:40.628167 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "176dbb86bc6448a8b959aa576c60a6e8"
format_stamp: "Formatted at 2026-09-27 13:47:40 on dist-test-slave-7g3l"
server_key: "9d36c98c13dae3b58274ae09ee75be80"
server_key_iv: "819da8ffe01665b0966397d325ab14bd"
server_key_version: "encryptionkey@0"
I20260927 13:47:40.628315 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:40.647254 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:40.650812 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:40.651053 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:40.668527 20178 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:34863
I20260927 13:47:40.668560 20244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:34863 every 8 connection(s)
I20260927 13:47:40.669795 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:40.673333 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:40.677616 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20178
I20260927 13:47:40.677791 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:40.678215 1975 external_mini_cluster.cc:1428] Setting key b71ce3a639f0c99fa85e8423c45f94aa
I20260927 13:47:40.685034 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: Bootstrap starting.
I20260927 13:47:40.688426 20245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:40.689566 20245 log.cc:824] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:40.692315 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: No bootstrap required, opened a new log
Sep 27 13:47:40 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:40.702288 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:40.679902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.1.237.254:34863)
0927 13:47:40.680699 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:40.680718 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:40.680765 (+ 47us) server_negotiation.cc:407] Connection header received
0927 13:47:40.683323 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:40.683364 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:40.683806 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:40.684266 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:40.685528 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:40.687484 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:40.688789 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:40.689793 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:40.689959 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:40.693960 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:40.693970 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:40.693987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:40.694040 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:40.698047 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:40.698650 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:40.698664 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:40.698672 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:40.698878 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:40.699207 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:40.699215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:40.699219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:40.699904 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:40.700165 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:40.700536 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 13:47:40.700988 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":314,"threads_started":1}
I20260927 13:47:40.702785 20245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } }
I20260927 13:47:40.703130 20245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:40.703262 20245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176dbb86bc6448a8b959aa576c60a6e8, State: Initialized, Role: FOLLOWER
I20260927 13:47:40.703850 20245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } }
I20260927 13:47:40.704097 20245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:40.704233 20245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:40.704407 20245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:40.706512 20245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } }
I20260927 13:47:40.706954 20245 leader_election.cc:302] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 176dbb86bc6448a8b959aa576c60a6e8; no voters:
I20260927 13:47:40.707541 20245 leader_election.cc:288] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:40.707681 20250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:40.708312 20250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [term 1 LEADER]: Becoming Leader. State: Replica: 176dbb86bc6448a8b959aa576c60a6e8, State: Running, Role: LEADER
I20260927 13:47:40.708815 20250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } }
I20260927 13:47:40.709438 20245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:40.711630 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176dbb86bc6448a8b959aa576c60a6e8. Latest consensus state: current_term: 1 leader_uuid: "176dbb86bc6448a8b959aa576c60a6e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } } }
I20260927 13:47:40.711700 20251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "176dbb86bc6448a8b959aa576c60a6e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176dbb86bc6448a8b959aa576c60a6e8" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34863 } } }
I20260927 13:47:40.712013 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:40.712062 20251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:40.712967 20257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:40.719012 20257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:40.726959 20257 catalog_manager.cc:1347] Generated new cluster ID: 96574f18d6fe43d6bcd127237c9cfca3
I20260927 13:47:40.727066 20257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:40.744902 20257 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:40.746134 20257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:40.754133 20257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: Generated new TSK 0
I20260927 13:47:40.754884 20257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:47:40.811693 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:41.130903 20271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:41.131313 20271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:41.131487 20271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:41.138809 20271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:41.138926 20271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:41.139097 20271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:41.148577 20271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:41.149698 20271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:41.150955 20271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:41.163066 20276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:41.163323 20279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:41.163345 20277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:41.164043 20271 server_base.cc:1034] running on GCE node
I20260927 13:47:41.164569 20271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:41.165385 20271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:41.166608 20271 hybrid_clock.cc:648] HybridClock initialized: now 1790516861166570 us; error 35 us; skew 500 ppm
I20260927 13:47:41.166942 20271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:41 dist-test-slave-7g3l krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516861, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:41.170336 20271 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:47:41.171969 20271 webserver.cc:460] Webserver started at http://127.1.237.193:33567/ using document root <none> and password file <none>
I20260927 13:47:41.172751 20271 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:41.172881 20271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:41.173182 20271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:41.175863 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "6e7d20a3c5a94e908f88b963b41e341e"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "333d27d996ae1537c8c7989aeb655289"
server_key_iv: "a386bf313381e9e1acbbcb7cd38df938"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.176573 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "6e7d20a3c5a94e908f88b963b41e341e"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "333d27d996ae1537c8c7989aeb655289"
server_key_iv: "a386bf313381e9e1acbbcb7cd38df938"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.181146 20271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:47:41.184430 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.185629 20271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:47:41.185796 20271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "6e7d20a3c5a94e908f88b963b41e341e"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "333d27d996ae1537c8c7989aeb655289"
server_key_iv: "a386bf313381e9e1acbbcb7cd38df938"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.185979 20271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:41.211542 20271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:41.215301 20271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:41.215555 20271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:41.216383 20271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:41.217561 20271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:41.217659 20271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.217799 20271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:41.217862 20271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.249569 20271 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37403
I20260927 13:47:41.249642 20398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37403 every 8 connection(s)
I20260927 13:47:41.250759 20271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:41.253585 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20271
I20260927 13:47:41.253768 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:41.254293 1975 external_mini_cluster.cc:1428] Setting key 19170df3bc843f1de2edb2b0c14f78a3
Sep 27 13:47:41 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516861, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:41.270694 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:41.253196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:40831 (local address 127.1.237.254:34863)
0927 13:47:41.255869 (+ 2673us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:41.255891 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:41.255908 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:47:41.257317 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:41.257325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:41.257411 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:41.257525 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:41.258890 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.259976 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.261366 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.262047 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.262180 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:41.265345 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:41.265350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:41.265353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:41.265382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:41.267913 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:41.268573 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:41.268581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:41.268586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:41.268702 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:41.269109 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:41.269115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:41.269118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:41.269356 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:41.269500 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:41.270135 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 13:47:41.270341 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2568,"thread_start_us":2462,"threads_started":1}
I20260927 13:47:41.271605 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:41.253897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34863 (local address 127.1.237.193:40831)
0927 13:47:41.255008 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:41.255059 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:41.257068 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:41.257721 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:41.257728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:41.258211 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:41.258730 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:41.258746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.260108 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.260111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:41.261231 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:41.261242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.262240 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.262244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:41.262424 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:41.263037 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:41.263058 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:41.265133 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:41.268042 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:41.268049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:41.268062 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:41.268458 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:41.268815 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:41.268820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:41.268823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:41.268998 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:41.269475 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:41.269482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:41.269789 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 13:47:41.270028 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 13:47:41.270483 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":556,"threads_started":1}
I20260927 13:47:41.273103 20399 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34863
I20260927 13:47:41.273361 20399 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:41.273988 20399 heartbeater.cc:511] Master 127.1.237.254:34863 requested a full tablet report, sending...
I20260927 13:47:41.276192 20210 ts_manager.cc:194] Registered new tserver with Master: 6e7d20a3c5a94e908f88b963b41e341e (127.1.237.193:37403)
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
I20260927 13:47:41.278338 20210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:40831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:47:41.304137 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:41.611330 20405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:41.611605 20405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:41.611696 20405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:41.618363 20405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:41.618465 20405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:41.618577 20405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:41.627372 20405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:41.628360 20405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:41.629459 20405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:41.641939 20410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:41.641899 20413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:41.641968 20411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:41.642719 20405 server_base.cc:1034] running on GCE node
I20260927 13:47:41.643313 20405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:41.644192 20405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:41.645378 20405 hybrid_clock.cc:648] HybridClock initialized: now 1790516861645336 us; error 33 us; skew 500 ppm
I20260927 13:47:41.645661 20405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:41 dist-test-slave-7g3l krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516861, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:41.651638 20405 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:47:41.653053 20405 webserver.cc:460] Webserver started at http://127.1.237.194:36947/ using document root <none> and password file <none>
I20260927 13:47:41.653695 20405 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:41.653831 20405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:41.654132 20405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:41.656939 20405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "57e56d048bfe47c7af0b9ee1d51c1680"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "8e3fae41f84098277e983f55af7ef9c3"
server_key_iv: "a94c372ed06240c13d5ffd37481219f7"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.657672 20405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "57e56d048bfe47c7af0b9ee1d51c1680"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "8e3fae41f84098277e983f55af7ef9c3"
server_key_iv: "a94c372ed06240c13d5ffd37481219f7"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.662343 20405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:47:41.665458 20420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.666654 20405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:47:41.666823 20405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "57e56d048bfe47c7af0b9ee1d51c1680"
format_stamp: "Formatted at 2026-09-27 13:47:41 on dist-test-slave-7g3l"
server_key: "8e3fae41f84098277e983f55af7ef9c3"
server_key_iv: "a94c372ed06240c13d5ffd37481219f7"
server_key_version: "encryptionkey@0"
I20260927 13:47:41.667012 20405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:41.686057 20405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:41.690618 20405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:41.690829 20405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:41.691689 20405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:41.692826 20405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:41.692898 20405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.692978 20405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:41.693022 20405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:41.724530 20405 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:39507
I20260927 13:47:41.724577 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:39507 every 8 connection(s)
I20260927 13:47:41.725759 20405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:41.732765 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20405
I20260927 13:47:41.732949 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:41.733435 1975 external_mini_cluster.cc:1428] Setting key a415846bd26ab20d54b2157f8554d3e9
Sep 27 13:47:41 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516861, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:41.746170 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:41.728198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:41525 (local address 127.1.237.254:34863)
0927 13:47:41.728406 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:41.728412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:41.729447 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 13:47:41.731086 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:41.731090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:41.731199 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:41.731328 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:41.732772 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.733774 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.735806 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.736525 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.736659 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:41.739725 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:41.739734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:41.739742 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:41.739784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:41.743123 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:41.744071 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:41.744075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:41.744077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:41.744199 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:41.744685 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:41.744688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:41.744690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:41.744920 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:41.745030 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:41.745796 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 13:47:41.745943 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 13:47:41.747558 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:41.728635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34863 (local address 127.1.237.194:41525)
0927 13:47:41.729303 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:41.729338 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:41.730798 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:41.731546 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:41.731554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:41.732032 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:41.732606 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:41.732623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.733936 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.733942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:41.735677 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:41.735689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:41.736717 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:41.736721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:41.736898 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:41.737515 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:41.737535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:41.739531 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:41.743289 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:41.743300 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:41.743314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:41.743926 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:41.744312 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:41.744320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:41.744326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:41.744573 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:41.745040 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:41.745051 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:41.745460 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 13:47:41.745734 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 13:47:41.746230 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":270,"threads_started":1}
I20260927 13:47:41.749070 20533 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34863
I20260927 13:47:41.749351 20533 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:41.749979 20533 heartbeater.cc:511] Master 127.1.237.254:34863 requested a full tablet report, sending...
I20260927 13:47:41.751369 20210 ts_manager.cc:194] Registered new tserver with Master: 57e56d048bfe47c7af0b9ee1d51c1680 (127.1.237.194:39507)
I20260927 13:47:41.752607 20210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:41525
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:47:41.784298 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:42.095806 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:42.096062 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:42.096153 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:42.102990 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:42.103080 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:42.103206 20538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:42.112361 20538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:42.113389 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:42.114568 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:42.126897 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:42.126942 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:42.126971 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:42.127719 20538 server_base.cc:1034] running on GCE node
I20260927 13:47:42.128273 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:42.129086 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:42.130298 20538 hybrid_clock.cc:648] HybridClock initialized: now 1790516862130259 us; error 50 us; skew 500 ppm
I20260927 13:47:42.130580 20538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:42 dist-test-slave-7g3l krb5kdc[20165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:42.133540 20538 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:47:42.134789 20538 webserver.cc:460] Webserver started at http://127.1.237.195:46649/ using document root <none> and password file <none>
I20260927 13:47:42.135497 20538 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:42.135632 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:42.135915 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:42.138752 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "d6d56e8e618945a59c2d2206eb681149"
format_stamp: "Formatted at 2026-09-27 13:47:42 on dist-test-slave-7g3l"
server_key: "73ebf1978729c77a7adbb4d311e51dec"
server_key_iv: "9e0e9c93962820d1cff84a87ea1e749e"
server_key_version: "encryptionkey@0"
I20260927 13:47:42.139544 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "d6d56e8e618945a59c2d2206eb681149"
format_stamp: "Formatted at 2026-09-27 13:47:42 on dist-test-slave-7g3l"
server_key: "73ebf1978729c77a7adbb4d311e51dec"
server_key_iv: "9e0e9c93962820d1cff84a87ea1e749e"
server_key_version: "encryptionkey@0"
I20260927 13:47:42.144126 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:47:42.147241 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:42.148401 20538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:47:42.148567 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "d6d56e8e618945a59c2d2206eb681149"
format_stamp: "Formatted at 2026-09-27 13:47:42 on dist-test-slave-7g3l"
server_key: "73ebf1978729c77a7adbb4d311e51dec"
server_key_iv: "9e0e9c93962820d1cff84a87ea1e749e"
server_key_version: "encryptionkey@0"
I20260927 13:47:42.148747 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:42.168808 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:42.172380 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:42.172672 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:42.173830 20538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:42.175434 20538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:42.175531 20538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:42.175644 20538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:42.175721 20538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:42.208552 20538 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:34987
I20260927 13:47:42.208632 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:34987 every 8 connection(s)
I20260927 13:47:42.209929 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:42.213256 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20538
I20260927 13:47:42.213440 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:42.214056 1975 external_mini_cluster.cc:1428] Setting key 59c1dbbdad03ed5050f19ef93bcf37c6
Sep 27 13:47:42 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:42.229323 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.212730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:59021 (local address 127.1.237.254:34863)
0927 13:47:42.212915 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.212919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.214330 (+ 1411us) server_negotiation.cc:407] Connection header received
0927 13:47:42.216716 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.216721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.216804 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.216911 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:42.218295 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.218982 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.220433 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.220827 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.220926 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.223892 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:42.223897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:42.223900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:42.223928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:42.226452 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.227083 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.227087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.227089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.227252 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.227681 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.227686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.227689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.228015 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:42.228166 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.228864 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.229091 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:42.230193 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.213214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:34863 (local address 127.1.237.195:59021)
0927 13:47:42.214159 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.214209 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.216436 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.217122 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.217130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.217611 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:42.218157 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.218177 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.219103 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.219106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.220307 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.220318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.221008 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.221012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.221192 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.221730 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:42.221753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:42.223721 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:42.226571 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:42.226577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:42.226587 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:42.226944 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:42.227381 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:42.227385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:42.227387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:42.227556 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:42.228179 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:42.228188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:42.228519 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.228769 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.229160 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
I20260927 13:47:42.231494 20666 heartbeater.cc:347] Connected to a master server at 127.1.237.254:34863
I20260927 13:47:42.231794 20666 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:42.232481 20666 heartbeater.cc:511] Master 127.1.237.254:34863 requested a full tablet report, sending...
I20260927 13:47:42.233680 20210 ts_manager.cc:194] Registered new tserver with Master: d6d56e8e618945a59c2d2206eb681149 (127.1.237.195:34987)
I20260927 13:47:42.234460 20210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:59021
I20260927 13:47:42.240860 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:42.257740 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.244695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.1.237.254:34863)
0927 13:47:42.244849 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.244853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.245302 (+ 449us) server_negotiation.cc:407] Connection header received
0927 13:47:42.245422 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.245426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.245489 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.245566 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:42.246869 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.247603 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.248924 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.249520 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.249634 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.252416 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:42.252421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:42.252425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:42.252459 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:42.255558 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.256220 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.256223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.256225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.256334 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.256667 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.256669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.256671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.256943 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:42.257144 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.257430 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.257543 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:47:42.261204 20210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:42.267938 20210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:42.280998 20399 heartbeater.cc:503] Master 127.1.237.254:34863 was elected leader, sending a full tablet report...
I20260927 13:47:42.289659 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.279229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.1.237.193:37403)
0927 13:47:42.281504 (+ 2275us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.281514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.281543 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:47:42.281611 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.281629 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.281901 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.282790 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:42.283049 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.284594 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.287197 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.288564 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.288710 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.288942 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.289217 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.289357 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2122,"thread_start_us":1953,"threads_started":1}
I20260927 13:47:42.289692 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.278967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37403 (local address 127.0.0.1:58220)
0927 13:47:42.279888 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.279929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.280091 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.282135 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.282142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.282165 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:42.282598 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.282615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.284690 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.284697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.287020 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.287040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.288752 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.288758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.288940 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.288995 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.289133 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.289299 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":1434,"threads_started":1}
I20260927 13:47:42.290794 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.281507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:39507 (local address 127.0.0.1:45634)
0927 13:47:42.283364 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.283386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.283510 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.284395 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.284400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.284427 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:42.284887 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.284903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.286723 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.286729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.288281 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.288296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.290083 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.290090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.290262 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.290295 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.290383 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.290485 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"thread_start_us":267,"threads_started":1}
I20260927 13:47:42.290966 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.281744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.1.237.194:39507)
0927 13:47:42.283986 (+ 2242us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.283991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.284019 (+ 28us) server_negotiation.cc:407] Connection header received
0927 13:47:42.284092 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.284096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.284226 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.284403 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:42.285073 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.286568 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.288506 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.289895 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.290028 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.290202 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.290485 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.290630 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2112,"thread_start_us":2005,"threads_started":1}
I20260927 13:47:42.291956 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.279203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34987 (local address 127.0.0.1:40740)
0927 13:47:42.282028 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.282047 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.282146 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.284092 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.284097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.284120 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:42.284459 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.284468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.285891 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.285898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.288137 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.288151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.290359 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.291177 (+ 818us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.291366 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.291396 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.291490 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.291619 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2697,"mutex_wait_us":3,"thread_start_us":1028,"threads_started":1}
I20260927 13:47:42.292783 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.279927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.1.237.195:34987)
0927 13:47:42.280976 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.281001 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.282261 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 13:47:42.282362 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.282368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.283687 (+ 1319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.283847 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:42.284682 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.285767 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.288336 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.290100 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.290206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.290428 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.292395 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.292553 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":638,"threads_started":1}
I20260927 13:47:42.293637 20334 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (DEFAULT_TABLE table=test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:42.295086 20334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf7d682546749e8a837eb4e3ecdd2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:42.294929 20468 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (DEFAULT_TABLE table=test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:42.295399 20601 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (DEFAULT_TABLE table=test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:42.296257 20468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf7d682546749e8a837eb4e3ecdd2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:42.296588 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf7d682546749e8a837eb4e3ecdd2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:42.306597 20684 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Bootstrap starting.
I20260927 13:47:42.308954 20685 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Bootstrap starting.
I20260927 13:47:42.310729 20684 tablet_bootstrap.cc:654] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:42.311415 20686 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Bootstrap starting.
I20260927 13:47:42.312209 20684 log.cc:824] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:42.312930 20685 tablet_bootstrap.cc:654] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:42.314023 20685 log.cc:824] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:42.314543 20686 tablet_bootstrap.cc:654] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:42.315328 20684 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: No bootstrap required, opened a new log
I20260927 13:47:42.315610 20684 ts_tablet_manager.cc:1397] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 13:47:42.315752 20686 log.cc:824] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:42.317070 20685 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: No bootstrap required, opened a new log
I20260927 13:47:42.317305 20685 ts_tablet_manager.cc:1397] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:47:42.318048 20686 tablet_bootstrap.cc:492] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: No bootstrap required, opened a new log
I20260927 13:47:42.318343 20686 ts_tablet_manager.cc:1397] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260927 13:47:42.325522 20686 raft_consensus.cc:348] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.325865 20686 raft_consensus.cc:374] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:42.326007 20686 raft_consensus.cc:729] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e56d048bfe47c7af0b9ee1d51c1680, State: Initialized, Role: FOLLOWER
I20260927 13:47:42.325917 20684 raft_consensus.cc:348] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.326320 20684 raft_consensus.cc:374] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:42.326414 20684 raft_consensus.cc:729] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d56e8e618945a59c2d2206eb681149, State: Initialized, Role: FOLLOWER
I20260927 13:47:42.326642 20686 consensus_queue.cc:260] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.326970 20684 consensus_queue.cc:260] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.327291 20685 raft_consensus.cc:348] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.327785 20685 raft_consensus.cc:374] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:42.327911 20685 raft_consensus.cc:729] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7d20a3c5a94e908f88b963b41e341e, State: Initialized, Role: FOLLOWER
I20260927 13:47:42.327939 20533 heartbeater.cc:503] Master 127.1.237.254:34863 was elected leader, sending a full tablet report...
I20260927 13:47:42.328536 20686 ts_tablet_manager.cc:1428] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260927 13:47:42.328656 20685 consensus_queue.cc:260] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.334040 20685 ts_tablet_manager.cc:1428] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260927 13:47:42.334935 20666 heartbeater.cc:503] Master 127.1.237.254:34863 was elected leader, sending a full tablet report...
I20260927 13:47:42.336553 20684 ts_tablet_manager.cc:1428] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260927 13:47:42.417006 20692 raft_consensus.cc:482] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:42.417256 20692 raft_consensus.cc:504] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.418805 20692 leader_election.cc:288] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d56e8e618945a59c2d2206eb681149 (127.1.237.195:34987), 57e56d048bfe47c7af0b9ee1d51c1680 (127.1.237.194:39507)
I20260927 13:47:42.425491 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.419295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:47343 (local address 127.1.237.195:34987)
0927 13:47:42.419503 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.419511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.420250 (+ 739us) server_negotiation.cc:407] Connection header received
0927 13:47:42.420414 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.420419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.420505 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.420597 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:42.421219 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.421998 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.423793 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.424568 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.424721 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.424810 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.425107 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.425232 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:42.425509 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.419406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:39211 (local address 127.1.237.194:39507)
0927 13:47:42.419631 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.419640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.420437 (+ 797us) server_negotiation.cc:407] Connection header received
0927 13:47:42.420636 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.420641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.420720 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.420823 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:42.421395 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.422098 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.423851 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.424594 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.424763 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.424834 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.425145 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.425262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 13:47:42.425633 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.419360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:39507 (local address 127.1.237.193:39211)
0927 13:47:42.420310 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.420342 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.420474 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.420844 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.420847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.420865 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:42.421218 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.421229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.422228 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.422232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.423691 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.423703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.424836 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.424844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.424971 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.424992 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.425100 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.425218 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":20,"thread_start_us":550,"threads_started":1}
I20260927 13:47:42.425880 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.419310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:34987 (local address 127.1.237.193:47343)
0927 13:47:42.420145 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:42.420160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:42.420274 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:42.420629 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:42.420633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:42.420658 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:42.421068 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.421080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.422124 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.422132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.423646 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:42.423661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.424791 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.424798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:42.424949 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.424972 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 13:47:42.425047 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 13:47:42.425183 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":18,"thread_start_us":460,"threads_started":1}
I20260927 13:47:42.426635 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf7d682546749e8a837eb4e3ecdd2b5" candidate_uuid: "6e7d20a3c5a94e908f88b963b41e341e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" is_pre_election: true
I20260927 13:47:42.426833 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf7d682546749e8a837eb4e3ecdd2b5" candidate_uuid: "6e7d20a3c5a94e908f88b963b41e341e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d56e8e618945a59c2d2206eb681149" is_pre_election: true
I20260927 13:47:42.427199 20488 raft_consensus.cc:2454] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7d20a3c5a94e908f88b963b41e341e in term 0.
I20260927 13:47:42.427367 20621 raft_consensus.cc:2454] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7d20a3c5a94e908f88b963b41e341e in term 0.
I20260927 13:47:42.427901 20288 leader_election.cc:302] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e56d048bfe47c7af0b9ee1d51c1680, 6e7d20a3c5a94e908f88b963b41e341e; no voters:
I20260927 13:47:42.428283 20692 raft_consensus.cc:2781] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:42.428434 20692 raft_consensus.cc:482] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:42.428567 20692 raft_consensus.cc:3037] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:42.430631 20692 raft_consensus.cc:504] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.431234 20692 leader_election.cc:288] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [CANDIDATE]: Term 1 election: Requested vote from peers d6d56e8e618945a59c2d2206eb681149 (127.1.237.195:34987), 57e56d048bfe47c7af0b9ee1d51c1680 (127.1.237.194:39507)
I20260927 13:47:42.431669 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf7d682546749e8a837eb4e3ecdd2b5" candidate_uuid: "6e7d20a3c5a94e908f88b963b41e341e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d56e8e618945a59c2d2206eb681149"
I20260927 13:47:42.431770 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf7d682546749e8a837eb4e3ecdd2b5" candidate_uuid: "6e7d20a3c5a94e908f88b963b41e341e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e56d048bfe47c7af0b9ee1d51c1680"
I20260927 13:47:42.431843 20621 raft_consensus.cc:3037] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:42.431993 20488 raft_consensus.cc:3037] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:42.434046 20621 raft_consensus.cc:2454] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7d20a3c5a94e908f88b963b41e341e in term 1.
I20260927 13:47:42.434109 20488 raft_consensus.cc:2454] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7d20a3c5a94e908f88b963b41e341e in term 1.
I20260927 13:47:42.434448 20290 leader_election.cc:302] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7d20a3c5a94e908f88b963b41e341e, d6d56e8e618945a59c2d2206eb681149; no voters:
I20260927 13:47:42.434744 20692 raft_consensus.cc:2781] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:42.435267 20692 raft_consensus.cc:686] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 1 LEADER]: Becoming Leader. State: Replica: 6e7d20a3c5a94e908f88b963b41e341e, State: Running, Role: LEADER
I20260927 13:47:42.435773 20692 consensus_queue.cc:237] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } }
I20260927 13:47:42.439939 20209 catalog_manager.cc:5495] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7d20a3c5a94e908f88b963b41e341e (127.1.237.193). New cstate: current_term: 1 leader_uuid: "6e7d20a3c5a94e908f88b963b41e341e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7d20a3c5a94e908f88b963b41e341e" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 } health_report { overall_health: UNKNOWN } } }
W20260927 13:47:42.461712 20667 tablet.cc:2369] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:42.477599 20534 tablet.cc:2369] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:42.483098 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.478144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.1.237.193:37403)
0927 13:47:42.478342 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.478348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.478442 (+ 94us) server_negotiation.cc:407] Connection header received
0927 13:47:42.478624 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.478627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.478698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.478836 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:42.479227 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.479892 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.481124 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.481505 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.481589 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.482075 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:42.482513 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:42.482640 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.482763 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.482823 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:42.490357 20488 raft_consensus.cc:1261] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 1 FOLLOWER]: Refusing update from remote peer 6e7d20a3c5a94e908f88b963b41e341e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:42.490448 20621 raft_consensus.cc:1261] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 1 FOLLOWER]: Refusing update from remote peer 6e7d20a3c5a94e908f88b963b41e341e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:42.491441 20692 consensus_queue.cc:1035] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e56d048bfe47c7af0b9ee1d51c1680" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:42.491714 20695 consensus_queue.cc:1035] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d56e8e618945a59c2d2206eb681149" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:42.502089 20698 mvcc.cc:205] Tried to move back new op lower bound from 7333957068751876096 to 7333957068540149760. Current Snapshot: MvccSnapshot[applied={T|T < 7333957068751876096}]
I20260927 13:47:42.503327 20699 mvcc.cc:205] Tried to move back new op lower bound from 7333957068751876096 to 7333957068540149760. Current Snapshot: MvccSnapshot[applied={T|T < 7333957068751876096}]
I20260927 13:47:42.514782 20210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:42.515113 20210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:42.518612 20210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 176dbb86bc6448a8b959aa576c60a6e8: Sending DeleteTablet for 3 replicas of tablet 6bf7d682546749e8a837eb4e3ecdd2b5
I20260927 13:47:42.519688 20601 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:42 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:40740
I20260927 13:47:42.519644 20334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:42 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:58220
I20260927 13:47:42.521443 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20271
I20260927 13:47:42.522632 20468 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf7d682546749e8a837eb4e3ecdd2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:42 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:45634
I20260927 13:47:42.533075 20708 tablet_replica.cc:330] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: stopping tablet replica
I20260927 13:47:42.534379 20708 raft_consensus.cc:2229] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:42.534531 20711 tablet_replica.cc:330] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: stopping tablet replica
I20260927 13:47:42.534871 20711 raft_consensus.cc:2229] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:42.534998 20708 raft_consensus.cc:2258] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:42.535353 20711 raft_consensus.cc:2258] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:42.535748 20709 tablet_replica.cc:330] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: stopping tablet replica
I20260927 13:47:42.536008 20709 raft_consensus.cc:2229] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:42.536309 20709 raft_consensus.cc:2258] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:42.538077 20708 ts_tablet_manager.cc:1905] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:42.540197 20711 ts_tablet_manager.cc:1905] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:42.541749 20709 ts_tablet_manager.cc:1905] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 13:47:42 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:47:42.546012 20708 ts_tablet_manager.cc:1918] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:42.546173 20708 log.cc:1196] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/6bf7d682546749e8a837eb4e3ecdd2b5
I20260927 13:47:42.546658 20708 ts_tablet_manager.cc:1939] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 6e7d20a3c5a94e908f88b963b41e341e: Deleting consensus metadata
I20260927 13:47:42.547125 20711 ts_tablet_manager.cc:1918] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:42.547278 20711 log.cc:1196] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/6bf7d682546749e8a837eb4e3ecdd2b5
I20260927 13:47:42.548974 20195 catalog_manager.cc:4841] TS 6e7d20a3c5a94e908f88b963b41e341e (127.1.237.193:37403): tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (table test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]) successfully deleted
I20260927 13:47:42.549350 20711 ts_tablet_manager.cc:1939] T 6bf7d682546749e8a837eb4e3ecdd2b5 P 57e56d048bfe47c7af0b9ee1d51c1680: Deleting consensus metadata
I20260927 13:47:42.551127 20195 catalog_manager.cc:4841] TS 57e56d048bfe47c7af0b9ee1d51c1680 (127.1.237.194:39507): tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (table test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]) successfully deleted
I20260927 13:47:42.553942 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.523025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.1.237.193:37403)
0927 13:47:42.523750 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.523759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.529255 (+ 5496us) server_negotiation.cc:407] Connection header received
0927 13:47:42.529449 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.529457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.529558 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.529665 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:42.535265 (+ 5600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.536446 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.538644 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.539302 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.539422 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.546906 (+ 7484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:42.546918 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:42.546939 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:42.546976 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:42.550740 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.551685 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.551693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.551697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.551834 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.552256 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.552261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.552264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.552713 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:42.553040 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.553343 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.553582 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552}
I20260927 13:47:42.554131 20709 ts_tablet_manager.cc:1918] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:42.554270 20709 log.cc:1196] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/6bf7d682546749e8a837eb4e3ecdd2b5
I20260927 13:47:42.554539 20394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:42.554772 20709 ts_tablet_manager.cc:1939] T 6bf7d682546749e8a837eb4e3ecdd2b5 P d6d56e8e618945a59c2d2206eb681149: Deleting consensus metadata
I20260927 13:47:42.556511 20197 catalog_manager.cc:4841] TS d6d56e8e618945a59c2d2206eb681149 (127.1.237.195:34987): tablet 6bf7d682546749e8a837eb4e3ecdd2b5 (table test-table [id=9ae7307294ec4a5e84dd7d16e845eff5]) successfully deleted
I20260927 13:47:42.647686 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20271
I20260927 13:47:42.666060 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20405
Sep 27 13:47:42 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:47:42.680562 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.666848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.1.237.194:39507)
0927 13:47:42.667018 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.667022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.667038 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:47:42.667092 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.667096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.667206 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.667325 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:42.668598 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.669336 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.670583 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.671014 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.671150 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.675492 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:42.675503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:42.675516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:42.675548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:42.678049 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.678666 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.678673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.678675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.678787 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.679167 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.679176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.679181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.679515 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:42.679808 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.680064 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.680262 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:47:42.681244 20528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:42.767212 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20405
I20260927 13:47:42.783473 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20538
Sep 27 13:47:42 dist-test-slave-7g3l krb5kdc[20165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:47:42.798306 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:42.784154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.1.237.195:34987)
0927 13:47:42.784342 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:42.784348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:42.784367 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:47:42.784539 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:42.784545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:42.784618 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:42.784727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:42.785984 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.786721 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.788116 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:42.788496 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:42.788628 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:42.793025 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:42.793037 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:42.793053 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:42.793085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:42.795630 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.796310 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.796319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.796321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.796453 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:42.796837 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:42.796843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:42.796846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:42.797199 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:42.797492 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:42.797784 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 13:47:42.798015 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 13:47:42.798933 20661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:42.884588 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20538
I20260927 13:47:42.900512 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20178
I20260927 13:47:42.901580 20240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:43.010308 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20178
2026-09-27T13:47:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4040 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T13:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:43Z Disabled control of system clock
I20260927 13:47:43.081811 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45885
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:43.390718 20719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:43.390969 20719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:43.391096 20719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:43.397760 20719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:43.397866 20719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:43.397949 20719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:43.398046 20719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:43.398118 20719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:43.406626 20719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:43.407716 20719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:43.409404 20719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:43.420634 20725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:43.420828 20727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:43.420668 20724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:43.421540 20719 server_base.cc:1034] running on GCE node
I20260927 13:47:43.422250 20719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:43.423645 20719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:43.424870 20719 hybrid_clock.cc:648] HybridClock initialized: now 1790516863424832 us; error 47 us; skew 500 ppm
I20260927 13:47:43.425207 20719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:43.433183 20719 webserver.cc:460] Webserver started at http://127.1.237.254:40149/ using document root <none> and password file <none>
I20260927 13:47:43.433893 20719 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:43.434017 20719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:43.434352 20719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:43.437348 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "e329ac4d40e144a3805e7febe26c2080"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "98a26a96a9fa281133654445e690c95c"
server_key_iv: "0222a516100857630182b35ee4a45d5c"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.438167 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "e329ac4d40e144a3805e7febe26c2080"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "98a26a96a9fa281133654445e690c95c"
server_key_iv: "0222a516100857630182b35ee4a45d5c"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.444739 20719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 13:47:43.448128 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:43.449352 20719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:47:43.449530 20719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "e329ac4d40e144a3805e7febe26c2080"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "98a26a96a9fa281133654445e690c95c"
server_key_iv: "0222a516100857630182b35ee4a45d5c"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.449728 20719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:43.462033 20719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:43.462770 20719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:43.463045 20719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:43.483297 20719 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:45885
I20260927 13:47:43.483385 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:45885 every 8 connection(s)
I20260927 13:47:43.484886 20719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:43.489683 20785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:43.490113 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20719
I20260927 13:47:43.490306 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:43.490885 1975 external_mini_cluster.cc:1428] Setting key b28840bc83d0023b194f6e6fccbae376
I20260927 13:47:43.506282 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: Bootstrap starting.
I20260927 13:47:43.507777 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:43.492894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.1.237.254:45885)
0927 13:47:43.494070 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:43.494087 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:43.494139 (+ 52us) server_negotiation.cc:407] Connection header received
0927 13:47:43.496124 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:43.496156 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:43.496722 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:43.497149 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:43.497743 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:43.499694 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:43.501829 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:43.502963 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:43.503197 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:43.504349 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:43.504362 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:43.504382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:43.505207 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:43.505326 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:43.505379 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:43.505502 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:43.505695 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 13:47:43.506235 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":442,"threads_started":1}
I20260927 13:47:43.510643 20785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:43.512058 20785 log.cc:824] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:43.515731 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: No bootstrap required, opened a new log
I20260927 13:47:43.527398 20785 raft_consensus.cc:348] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } }
I20260927 13:47:43.527748 20785 raft_consensus.cc:374] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:43.527817 20785 raft_consensus.cc:729] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e329ac4d40e144a3805e7febe26c2080, State: Initialized, Role: FOLLOWER
I20260927 13:47:43.528291 20785 consensus_queue.cc:260] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } }
I20260927 13:47:43.528508 20785 raft_consensus.cc:388] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:43.528605 20785 raft_consensus.cc:482] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:43.528719 20785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:43.530714 20785 raft_consensus.cc:504] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } }
I20260927 13:47:43.531144 20785 leader_election.cc:302] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e329ac4d40e144a3805e7febe26c2080; no voters:
I20260927 13:47:43.531755 20785 leader_election.cc:288] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:43.531944 20790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:43.532881 20790 raft_consensus.cc:686] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [term 1 LEADER]: Becoming Leader. State: Replica: e329ac4d40e144a3805e7febe26c2080, State: Running, Role: LEADER
I20260927 13:47:43.533335 20790 consensus_queue.cc:237] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } }
I20260927 13:47:43.533805 20785 sys_catalog.cc:564] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:43.536015 20791 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e329ac4d40e144a3805e7febe26c2080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } } }
I20260927 13:47:43.536409 20791 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:43.536128 20792 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e329ac4d40e144a3805e7febe26c2080. Latest consensus state: current_term: 1 leader_uuid: "e329ac4d40e144a3805e7febe26c2080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e329ac4d40e144a3805e7febe26c2080" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45885 } } }
I20260927 13:47:43.536525 20792 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:43.540069 20796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:43.544363 20796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:43.551946 20796 catalog_manager.cc:1347] Generated new cluster ID: e16ab390eb414e70bedbd2f816a1bc33
I20260927 13:47:43.552043 20796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:43.572795 20796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:43.573890 20796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:43.582017 20796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: Generated new TSK 0
I20260927 13:47:43.582732 20796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:43.594573 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--builtin_ntp_servers=127.1.237.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:43.904696 20809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:43.904942 20809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:43.905099 20809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:43.912005 20809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:43.912103 20809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:43.912307 20809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:43.920449 20809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:43.921572 20809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:43.923202 20809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:43.936055 20817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:43.936254 20815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:43.936373 20814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:43.937080 20809 server_base.cc:1034] running on GCE node
I20260927 13:47:43.937673 20809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:43.938644 20809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:43.939831 20809 hybrid_clock.cc:648] HybridClock initialized: now 1790516863939801 us; error 45 us; skew 500 ppm
I20260927 13:47:43.940131 20809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:43.942158 20809 webserver.cc:460] Webserver started at http://127.1.237.193:46135/ using document root <none> and password file <none>
I20260927 13:47:43.942763 20809 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:43.942867 20809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:43.943220 20809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:43.945817 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "2f696adad7c64370b28d013b5e661179"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "346d22609aa968bda8f4aec20dbc4bd4"
server_key_iv: "8ad32c8f8a33f932381d9217a1808c90"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.946533 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "2f696adad7c64370b28d013b5e661179"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "346d22609aa968bda8f4aec20dbc4bd4"
server_key_iv: "8ad32c8f8a33f932381d9217a1808c90"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.951011 20809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 13:47:43.954306 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:43.955713 20809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:43.955884 20809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "2f696adad7c64370b28d013b5e661179"
format_stamp: "Formatted at 2026-09-27 13:47:43 on dist-test-slave-7g3l"
server_key: "346d22609aa968bda8f4aec20dbc4bd4"
server_key_iv: "8ad32c8f8a33f932381d9217a1808c90"
server_key_version: "encryptionkey@0"
I20260927 13:47:43.956096 20809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:43.969969 20809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:43.970647 20809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:43.970938 20809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:43.971849 20809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:43.973111 20809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:43.973225 20809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:43.973352 20809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:43.973436 20809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.005394 20809 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:37359
I20260927 13:47:44.005477 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:37359 every 8 connection(s)
I20260927 13:47:44.006671 20809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:44.008522 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20809
I20260927 13:47:44.008690 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:44.009261 1975 external_mini_cluster.cc:1428] Setting key 1e47084ab083429782de84e8279661fe
I20260927 13:47:44.013276 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--builtin_ntp_servers=127.1.237.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:44.019624 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.009228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:55487 (local address 127.1.237.254:45885)
0927 13:47:44.011536 (+ 2308us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.011542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.011563 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:47:44.013192 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.013197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.013286 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.013404 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:44.014568 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.015296 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.016941 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.017351 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.017476 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.018796 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:44.018801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:44.018804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:44.018820 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:44.018856 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:44.018867 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:44.018959 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.019210 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.019301 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2185,"thread_start_us":2085,"threads_started":1}
I20260927 13:47:44.020521 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.009978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45885 (local address 127.1.237.193:55487)
0927 13:47:44.010971 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.011053 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.012912 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.013751 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.013759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.013853 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:44.014405 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.014423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.015435 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.015440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.016792 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.016804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.017567 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.017575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.017798 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.018491 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:44.018495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:44.018534 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:44.018603 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:44.018958 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:44.018971 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:44.018980 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.019198 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.019556 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":325,"threads_started":1}
I20260927 13:47:44.021893 20936 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45885
I20260927 13:47:44.022174 20936 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:44.022977 20936 heartbeater.cc:511] Master 127.1.237.254:45885 requested a full tablet report, sending...
I20260927 13:47:44.025161 20750 ts_manager.cc:194] Registered new tserver with Master: 2f696adad7c64370b28d013b5e661179 (127.1.237.193:37359)
I20260927 13:47:44.027320 20750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.193:55487
W20260927 13:47:44.324795 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:44.325027 20940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:44.325114 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:44.331681 20940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:44.331759 20940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:44.331853 20940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:44.340724 20940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:44.341835 20940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:44.343627 20940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:44.356096 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:44.356181 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:44.356141 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:44.357023 20940 server_base.cc:1034] running on GCE node
I20260927 13:47:44.357575 20940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:44.358423 20940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:44.359648 20940 hybrid_clock.cc:648] HybridClock initialized: now 1790516864359592 us; error 44 us; skew 500 ppm
I20260927 13:47:44.359961 20940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:44.362046 20940 webserver.cc:460] Webserver started at http://127.1.237.194:34765/ using document root <none> and password file <none>
I20260927 13:47:44.362675 20940 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:44.362789 20940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:44.363112 20940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:44.365741 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "dd7968c023fc4a8b903b7f4dda19d0d0"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "43c0a842e9f6cfb2c026b1f648ea688b"
server_key_iv: "24471966c7f5c1d51acda1c8598fd8b2"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.366497 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "dd7968c023fc4a8b903b7f4dda19d0d0"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "43c0a842e9f6cfb2c026b1f648ea688b"
server_key_iv: "24471966c7f5c1d51acda1c8598fd8b2"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.370996 20940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:47:44.374327 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.375553 20940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:44.375707 20940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "dd7968c023fc4a8b903b7f4dda19d0d0"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "43c0a842e9f6cfb2c026b1f648ea688b"
server_key_iv: "24471966c7f5c1d51acda1c8598fd8b2"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.375918 20940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:44.432399 20940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:44.433218 20940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:44.433509 20940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:44.434566 20940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:44.436180 20940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:44.436285 20940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.436414 20940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:44.436501 20940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.468631 20940 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:35777
I20260927 13:47:44.468721 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:35777 every 8 connection(s)
I20260927 13:47:44.469866 20940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:44.477907 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 20940
I20260927 13:47:44.478073 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:44.478495 1975 external_mini_cluster.cc:1428] Setting key 69ea8268c3dce598ea0c9bdc62c042a1
I20260927 13:47:44.481563 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--builtin_ntp_servers=127.1.237.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:44.481673 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.472260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:40419 (local address 127.1.237.254:45885)
0927 13:47:44.472434 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.472438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.473714 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 13:47:44.475377 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.475381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.475447 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.475536 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:44.476458 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.477129 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.478680 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.479293 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.479420 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.480852 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:44.480859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:44.480863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:44.480885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:44.480922 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:44.480937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:44.481055 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.481250 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.481349 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:47:44.482733 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.472771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45885 (local address 127.1.237.194:40419)
0927 13:47:44.473580 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.473619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.475082 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.475716 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.475724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.475810 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:44.476327 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.476343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.477245 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.477248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.478524 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.478545 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.479473 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.479478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.479778 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.480550 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:44.480553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:44.480612 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:44.480681 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:44.481047 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:44.481053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:44.481058 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.481208 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.481444 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":245,"threads_started":1}
I20260927 13:47:44.491943 21067 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45885
I20260927 13:47:44.492242 21067 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:44.493057 21067 heartbeater.cc:511] Master 127.1.237.254:45885 requested a full tablet report, sending...
I20260927 13:47:44.494561 20750 ts_manager.cc:194] Registered new tserver with Master: dd7968c023fc4a8b903b7f4dda19d0d0 (127.1.237.194:35777)
I20260927 13:47:44.495795 20750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.194:40419
W20260927 13:47:44.799015 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:44.799329 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:44.799499 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:44.806440 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:44.806524 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:44.806723 21070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:44.815635 21070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:44.816830 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:44.818589 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:44.831173 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:44.831357 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:44.831211 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:44.831847 21070 server_base.cc:1034] running on GCE node
I20260927 13:47:44.832423 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:44.833269 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:44.834461 21070 hybrid_clock.cc:648] HybridClock initialized: now 1790516864834398 us; error 55 us; skew 500 ppm
I20260927 13:47:44.834765 21070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:44.836896 21070 webserver.cc:460] Webserver started at http://127.1.237.195:39621/ using document root <none> and password file <none>
I20260927 13:47:44.837527 21070 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:44.837654 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:44.838034 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:44.840894 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "935e4d26f8854b4fb04693a80e19be2a"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "725a2ba80b7856ad635971a99fb37946"
server_key_iv: "2c3568b9cbf750e19fb07b0ab4eba9e8"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.841641 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "935e4d26f8854b4fb04693a80e19be2a"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "725a2ba80b7856ad635971a99fb37946"
server_key_iv: "2c3568b9cbf750e19fb07b0ab4eba9e8"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.846381 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 13:47:44.849619 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.850802 21070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:47:44.851020 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "935e4d26f8854b4fb04693a80e19be2a"
format_stamp: "Formatted at 2026-09-27 13:47:44 on dist-test-slave-7g3l"
server_key: "725a2ba80b7856ad635971a99fb37946"
server_key_iv: "2c3568b9cbf750e19fb07b0ab4eba9e8"
server_key_version: "encryptionkey@0"
I20260927 13:47:44.851277 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:44.876976 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:44.877658 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:44.877924 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:44.878844 21070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:44.880306 21070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:44.880422 21070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.880541 21070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:44.880621 21070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:44.912106 21070 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:45871
I20260927 13:47:44.912181 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:45871 every 8 connection(s)
I20260927 13:47:44.913314 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:44.914942 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21070
I20260927 13:47:44.915114 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:44.915576 1975 external_mini_cluster.cc:1428] Setting key 5870018221527c8749735b83b599536c
I20260927 13:47:44.924862 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.915933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:44895 (local address 127.1.237.254:45885)
0927 13:47:44.916129 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.916133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.917619 (+ 1486us) server_negotiation.cc:407] Connection header received
0927 13:47:44.919453 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.919457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.919518 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.919619 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:44.920547 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.921270 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.922638 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.923011 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.923106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.924237 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:44.924241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:44.924244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:44.924259 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:44.924286 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:44.924297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:44.924379 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.924554 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.924625 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 13:47:44.925734 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.916478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45885 (local address 127.1.237.195:44895)
0927 13:47:44.917510 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.917544 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.919238 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.919803 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.919810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.919885 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:44.920408 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.920422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.921384 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.921387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.922528 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.922538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.923233 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.923238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.923402 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.924006 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:44.924008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:44.924040 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:44.924102 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:44.924394 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:44.924401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:44.924406 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.924529 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.924760 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":342,"threads_started":1}
I20260927 13:47:44.926898 21197 heartbeater.cc:347] Connected to a master server at 127.1.237.254:45885
I20260927 13:47:44.927192 21197 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:44.927819 21197 heartbeater.cc:511] Master 127.1.237.254:45885 requested a full tablet report, sending...
I20260927 13:47:44.929150 20750 ts_manager.cc:194] Registered new tserver with Master: 935e4d26f8854b4fb04693a80e19be2a (127.1.237.195:45871)
I20260927 13:47:44.930066 20750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.195:44895
I20260927 13:47:44.930289 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:44.940789 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.934943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.1.237.254:45885)
0927 13:47:44.935193 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.935198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.935501 (+ 303us) server_negotiation.cc:407] Connection header received
0927 13:47:44.935618 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.935621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.935698 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.935771 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:44.936211 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.936847 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.938390 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.938777 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.938866 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.940054 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:44.940059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:44.940062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:44.940078 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:44.940107 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:44.940119 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:44.940263 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.940453 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.940523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:47:44.944389 20750 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:44.951085 20750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:44.975931 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.963917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.1.237.195:45871)
0927 13:47:44.965187 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.965194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.967907 (+ 2713us) server_negotiation.cc:407] Connection header received
0927 13:47:44.968016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.968024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.968315 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.968535 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:44.969847 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.971362 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.973687 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.974569 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.974709 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.975043 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.975340 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.975545 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":878,"threads_started":1}
I20260927 13:47:44.976442 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.962877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:35777 (local address 127.0.0.1:51762)
0927 13:47:44.964003 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.964076 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.964264 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.968587 (+ 4323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.968595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.968627 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:44.969190 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.969207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.971098 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.971104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.973535 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.973553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.975459 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.975468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.975635 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.975669 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.975731 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.975913 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":64,"thread_start_us":4459,"threads_started":1}
I20260927 13:47:44.976861 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.963306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:45871 (local address 127.0.0.1:44138)
0927 13:47:44.967643 (+ 4337us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.967667 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.967794 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.968690 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.968697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.968725 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:44.969631 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.969645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.971534 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.971540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.973498 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.973510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.974760 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.974767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.975019 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.975078 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.976213 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.976351 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4183,"mutex_wait_us":60,"thread_start_us":954,"threads_started":1}
I20260927 13:47:44.978073 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.968609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52260)
0927 13:47:44.969219 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:44.969246 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:44.969361 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:44.970189 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:44.970192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:44.970210 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:44.970471 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.970478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.972505 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.972512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.974931 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:44.974964 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.977385 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.977390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:44.977502 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.977523 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 13:47:44.977625 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 13:47:44.977740 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":36,"thread_start_us":336,"threads_started":1}
I20260927 13:47:44.978206 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.968034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.1.237.193:37359)
0927 13:47:44.968607 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.968615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.969496 (+ 881us) server_negotiation.cc:407] Connection header received
0927 13:47:44.969595 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.969601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.969805 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.969976 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:44.970624 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.972121 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.975209 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.976921 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.977052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.977275 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.977675 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.977828 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260927 13:47:44.978353 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:44.963614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.1.237.194:35777)
0927 13:47:44.967957 (+ 4343us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:44.967963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:44.967985 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:47:44.968063 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:44.968068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:44.968315 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:44.968990 (+ 675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:44.969341 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.970939 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.973784 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:44.975240 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:44.976579 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:44.976862 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:44.976941 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 13:47:44.977132 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4171,"thread_start_us":4032,"threads_started":1}
I20260927 13:47:44.981264 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 (DEFAULT_TABLE table=test-table [id=3edcaefcefc04663b2e828c7e0eeb342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:44.981961 20871 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 (DEFAULT_TABLE table=test-table [id=3edcaefcefc04663b2e828c7e0eeb342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:44.982831 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9f26ff05bd41b195ede40015ec78b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:44.983349 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9f26ff05bd41b195ede40015ec78b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:44.985081 21002 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 (DEFAULT_TABLE table=test-table [id=3edcaefcefc04663b2e828c7e0eeb342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:44.986377 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9f26ff05bd41b195ede40015ec78b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:44.996676 21215 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Bootstrap starting.
I20260927 13:47:44.998747 21216 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Bootstrap starting.
I20260927 13:47:45.001624 21215 tablet_bootstrap.cc:654] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:45.002234 21216 tablet_bootstrap.cc:654] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:45.002573 21217 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Bootstrap starting.
I20260927 13:47:45.002916 21215 log.cc:824] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:45.003405 21216 log.cc:824] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:45.006256 21215 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: No bootstrap required, opened a new log
I20260927 13:47:45.006263 21216 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: No bootstrap required, opened a new log
I20260927 13:47:45.006435 21217 tablet_bootstrap.cc:654] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:45.006527 21215 ts_tablet_manager.cc:1397] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 13:47:45.006552 21216 ts_tablet_manager.cc:1397] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 13:47:45.007851 21217 log.cc:824] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:45.011713 21217 tablet_bootstrap.cc:492] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: No bootstrap required, opened a new log
I20260927 13:47:45.011981 21217 ts_tablet_manager.cc:1397] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 13:47:45.016937 21215 raft_consensus.cc:348] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.017378 21215 raft_consensus.cc:374] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:45.017477 21215 raft_consensus.cc:729] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7968c023fc4a8b903b7f4dda19d0d0, State: Initialized, Role: FOLLOWER
I20260927 13:47:45.018558 21215 consensus_queue.cc:260] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.021384 21216 raft_consensus.cc:348] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.021908 21216 raft_consensus.cc:374] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:45.022024 21216 raft_consensus.cc:729] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f696adad7c64370b28d013b5e661179, State: Initialized, Role: FOLLOWER
I20260927 13:47:45.021901 21217 raft_consensus.cc:348] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.022270 21217 raft_consensus.cc:374] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:45.022356 21217 raft_consensus.cc:729] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935e4d26f8854b4fb04693a80e19be2a, State: Initialized, Role: FOLLOWER
I20260927 13:47:45.022722 21216 consensus_queue.cc:260] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.022935 21217 consensus_queue.cc:260] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.025756 21067 heartbeater.cc:503] Master 127.1.237.254:45885 was elected leader, sending a full tablet report...
I20260927 13:47:45.026659 21215 ts_tablet_manager.cc:1428] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260927 13:47:45.027810 21197 heartbeater.cc:503] Master 127.1.237.254:45885 was elected leader, sending a full tablet report...
I20260927 13:47:45.028165 20936 heartbeater.cc:503] Master 127.1.237.254:45885 was elected leader, sending a full tablet report...
I20260927 13:47:45.028203 21217 ts_tablet_manager.cc:1428] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 13:47:45.028714 21216 ts_tablet_manager.cc:1428] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Time spent starting tablet: real 0.022s user 0.019s sys 0.000s
W20260927 13:47:45.164915 21198 tablet.cc:2369] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:45.196655 21222 raft_consensus.cc:482] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:45.196973 21222 raft_consensus.cc:504] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.198592 21222 leader_election.cc:288] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7968c023fc4a8b903b7f4dda19d0d0 (127.1.237.194:35777), 935e4d26f8854b4fb04693a80e19be2a (127.1.237.195:45871)
I20260927 13:47:45.207464 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.198993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:45067 (local address 127.1.237.195:45871)
0927 13:47:45.199375 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.199385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.199799 (+ 414us) server_negotiation.cc:407] Connection header received
0927 13:47:45.200035 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.200040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.200115 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.200216 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:45.201087 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.202212 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.205114 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.206316 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.206480 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.206545 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.207093 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.207233 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260927 13:47:45.207468 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.199112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:36041 (local address 127.1.237.194:35777)
0927 13:47:45.199395 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.199400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.199926 (+ 526us) server_negotiation.cc:407] Connection header received
0927 13:47:45.200055 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.200059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.200126 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.200216 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:45.201132 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.202212 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.205114 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.206316 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.206468 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.206545 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.207067 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.207211 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 13:47:45.207680 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.198982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:45871 (local address 127.1.237.193:45067)
0927 13:47:45.199692 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.199719 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:45.199879 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:45.200242 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:45.200247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:45.200285 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:45.200926 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:45.200941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.202335 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.202340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:45.204894 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:45.204908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.206554 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.206559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:45.206810 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.206929 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 13:47:45.207027 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 13:47:45.207225 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":350,"spinlock_wait_cycles":2688,"thread_start_us":419,"threads_started":1}
I20260927 13:47:45.208037 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.198997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:35777 (local address 127.1.237.193:36041)
0927 13:47:45.199827 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.199842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:45.199953 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:45.200242 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:45.200247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:45.200307 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:45.200964 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:45.200974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.202336 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.202340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:45.204893 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:45.204908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.206535 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.206543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:45.206807 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.206928 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 13:47:45.207027 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 13:47:45.207225 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":273,"spinlock_wait_cycles":46720,"thread_start_us":581,"threads_started":1}
I20260927 13:47:45.208751 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9f26ff05bd41b195ede40015ec78b6" candidate_uuid: "2f696adad7c64370b28d013b5e661179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935e4d26f8854b4fb04693a80e19be2a" is_pre_election: true
I20260927 13:47:45.208971 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9f26ff05bd41b195ede40015ec78b6" candidate_uuid: "2f696adad7c64370b28d013b5e661179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" is_pre_election: true
I20260927 13:47:45.209343 21152 raft_consensus.cc:2454] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f696adad7c64370b28d013b5e661179 in term 0.
I20260927 13:47:45.209455 21022 raft_consensus.cc:2454] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f696adad7c64370b28d013b5e661179 in term 0.
I20260927 13:47:45.210065 20824 leader_election.cc:302] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f696adad7c64370b28d013b5e661179, 935e4d26f8854b4fb04693a80e19be2a, dd7968c023fc4a8b903b7f4dda19d0d0; no voters:
I20260927 13:47:45.210476 21222 raft_consensus.cc:2781] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:45.210633 21222 raft_consensus.cc:482] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:45.210736 21222 raft_consensus.cc:3037] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:45.212899 21222 raft_consensus.cc:504] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
I20260927 13:47:45.213511 21222 leader_election.cc:288] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [CANDIDATE]: Term 1 election: Requested vote from peers dd7968c023fc4a8b903b7f4dda19d0d0 (127.1.237.194:35777), 935e4d26f8854b4fb04693a80e19be2a (127.1.237.195:45871)
I20260927 13:47:45.213932 21022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9f26ff05bd41b195ede40015ec78b6" candidate_uuid: "2f696adad7c64370b28d013b5e661179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0"
I20260927 13:47:45.214001 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9f26ff05bd41b195ede40015ec78b6" candidate_uuid: "2f696adad7c64370b28d013b5e661179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935e4d26f8854b4fb04693a80e19be2a"
I20260927 13:47:45.214141 21022 raft_consensus.cc:3037] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:45.214216 21152 raft_consensus.cc:3037] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:45.216500 21152 raft_consensus.cc:2454] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f696adad7c64370b28d013b5e661179 in term 1.
I20260927 13:47:45.216506 21022 raft_consensus.cc:2454] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f696adad7c64370b28d013b5e661179 in term 1.
I20260927 13:47:45.216910 20824 leader_election.cc:302] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f696adad7c64370b28d013b5e661179, 935e4d26f8854b4fb04693a80e19be2a; no voters:
I20260927 13:47:45.217183 21222 raft_consensus.cc:2781] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:45.217625 21222 raft_consensus.cc:686] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 1 LEADER]: Becoming Leader. State: Replica: 2f696adad7c64370b28d013b5e661179, State: Running, Role: LEADER
I20260927 13:47:45.218127 21222 consensus_queue.cc:237] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } }
W20260927 13:47:45.221755 21068 tablet.cc:2369] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:45.222544 20750 catalog_manager.cc:5495] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f696adad7c64370b28d013b5e661179 (127.1.237.193). New cstate: current_term: 1 leader_uuid: "2f696adad7c64370b28d013b5e661179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f696adad7c64370b28d013b5e661179" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 37359 } health_report { overall_health: HEALTHY } } }
W20260927 13:47:45.258951 20937 tablet.cc:2369] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:45.271364 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.266835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.1.237.193:37359)
0927 13:47:45.267025 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.267029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.267046 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:47:45.267231 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.267234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.267308 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.267414 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:45.267730 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.268372 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.269587 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.269958 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.270046 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.270480 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:45.270892 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:45.271039 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.271133 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.271184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:47:45.278661 21152 raft_consensus.cc:1261] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 1 FOLLOWER]: Refusing update from remote peer 2f696adad7c64370b28d013b5e661179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:45.278697 21022 raft_consensus.cc:1261] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 1 FOLLOWER]: Refusing update from remote peer 2f696adad7c64370b28d013b5e661179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:45.279559 21226 consensus_queue.cc:1035] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935e4d26f8854b4fb04693a80e19be2a" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:45.279883 21222 consensus_queue.cc:1035] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7968c023fc4a8b903b7f4dda19d0d0" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 35777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:45.291447 21230 mvcc.cc:205] Tried to move back new op lower bound from 7333957080172707840 to 7333957079937175552. Current Snapshot: MvccSnapshot[applied={T|T < 7333957080172707840}]
I20260927 13:47:45.293381 21231 mvcc.cc:205] Tried to move back new op lower bound from 7333957080172707840 to 7333957079937175552. Current Snapshot: MvccSnapshot[applied={T|T < 7333957080172707840}]
I20260927 13:47:45.302388 20749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:45.302642 20749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:45.306062 20749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e329ac4d40e144a3805e7febe26c2080: Sending DeleteTablet for 3 replicas of tablet 4b9f26ff05bd41b195ede40015ec78b6
I20260927 13:47:45.307025 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:45 UTC) from {username='slave'} at 127.0.0.1:44138
I20260927 13:47:45.307331 21002 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:45 UTC) from {username='slave'} at 127.0.0.1:51762
I20260927 13:47:45.307916 20871 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9f26ff05bd41b195ede40015ec78b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:45 UTC) from {username='slave'} at 127.0.0.1:52260
I20260927 13:47:45.308197 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20809
I20260927 13:47:45.316988 21239 tablet_replica.cc:330] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: stopping tablet replica
I20260927 13:47:45.317425 21239 raft_consensus.cc:2229] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:45.317920 21239 raft_consensus.cc:2258] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:45.321834 21239 ts_tablet_manager.cc:1905] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:45.325003 21240 tablet_replica.cc:330] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: stopping tablet replica
I20260927 13:47:45.325232 21238 tablet_replica.cc:330] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: stopping tablet replica
I20260927 13:47:45.325372 21240 raft_consensus.cc:2229] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:45.325704 21238 raft_consensus.cc:2229] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:45.325976 21240 raft_consensus.cc:2258] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:45.326216 21238 raft_consensus.cc:2258] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:45.328145 21238 ts_tablet_manager.cc:1905] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:45.328799 21239 ts_tablet_manager.cc:1918] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:45.328950 21239 log.cc:1196] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/4b9f26ff05bd41b195ede40015ec78b6
I20260927 13:47:45.329455 21240 ts_tablet_manager.cc:1905] T 4b9f26ff05bd41b195ede40015ec78b6 P 2f696adad7c64370b28d013b5e661179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:45.329530 21239 ts_tablet_manager.cc:1939] T 4b9f26ff05bd41b195ede40015ec78b6 P dd7968c023fc4a8b903b7f4dda19d0d0: Deleting consensus metadata
I20260927 13:47:45.331210 20736 catalog_manager.cc:4841] TS dd7968c023fc4a8b903b7f4dda19d0d0 (127.1.237.194:35777): tablet 4b9f26ff05bd41b195ede40015ec78b6 (table test-table [id=3edcaefcefc04663b2e828c7e0eeb342]) successfully deleted
I20260927 13:47:45.333029 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.309579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.1.237.193:37359)
0927 13:47:45.309793 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.309802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.320427 (+ 10625us) server_negotiation.cc:407] Connection header received
0927 13:47:45.320819 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.320824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.320918 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.320999 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:45.321605 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.322771 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.326305 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.326944 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.327038 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.331393 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:45.331409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:45.331423 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:45.332181 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:45.332239 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:45.332261 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:45.332336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.332561 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.332652 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:47:45.334106 20931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:45.337224 21238 ts_tablet_manager.cc:1918] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:45.337389 21238 log.cc:1196] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/4b9f26ff05bd41b195ede40015ec78b6
I20260927 13:47:45.337949 21238 ts_tablet_manager.cc:1939] T 4b9f26ff05bd41b195ede40015ec78b6 P 935e4d26f8854b4fb04693a80e19be2a: Deleting consensus metadata
I20260927 13:47:45.339326 20734 catalog_manager.cc:4841] TS 935e4d26f8854b4fb04693a80e19be2a (127.1.237.195:45871): tablet 4b9f26ff05bd41b195ede40015ec78b6 (table test-table [id=3edcaefcefc04663b2e828c7e0eeb342]) successfully deleted
I20260927 13:47:45.428125 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20809
W20260927 13:47:45.449098 20736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:37359 (error 108)
I20260927 13:47:45.449301 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20940
I20260927 13:47:45.451133 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.450699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52290)
0927 13:47:45.450872 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.451000 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 13:47:45.451557 20736 catalog_manager.cc:4584] TS 2f696adad7c64370b28d013b5e661179 (127.1.237.193:37359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b9f26ff05bd41b195ede40015ec78b6: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
I20260927 13:47:45.456621 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.450177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.1.237.194:35777)
0927 13:47:45.450366 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.450371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.450392 (+ 21us) server_negotiation.cc:407] Connection header received
0927 13:47:45.450538 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.450542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.450634 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.450753 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:45.451213 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.452155 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.453506 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.453876 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.454004 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.455284 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:45.455296 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:45.455310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:45.455891 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:45.455943 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:45.455963 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:45.456082 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.456201 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.456308 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:47:45.457273 21062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:45.505182 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.504758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52292)
0927 13:47:45.504960 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.505051 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:47:45.506167 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.505857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52294)
0927 13:47:45.505995 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.506064 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 13:47:45.544030 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20940
I20260927 13:47:45.561720 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21070
I20260927 13:47:45.566622 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.566139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52304)
0927 13:47:45.566329 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.566429 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:47:45.567839 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.567546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52310)
0927 13:47:45.567673 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.567743 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 13:47:45.568650 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.562448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.1.237.195:45871)
0927 13:47:45.562625 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:45.562633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:45.562658 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:47:45.562869 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:45.562873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:45.562959 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:45.563072 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:45.563475 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.564159 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.565457 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:45.565849 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:45.565996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:45.567091 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:45.567099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:45.567134 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:45.567861 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:45.567960 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:45.567988 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:45.568099 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:45.568240 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 13:47:45.568325 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:47:45.569244 21192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:45.648420 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.647870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52316)
0927 13:47:45.648142 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.648256 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 13:47:45.649533 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:45.649129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:37359 (local address 127.0.0.1:52330)
0927 13:47:45.649344 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:45.649409 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 13:47:45.657523 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21070
I20260927 13:47:45.673635 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 20719
I20260927 13:47:45.674634 20780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:45.778342 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 20719
2026-09-27T13:47:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2766 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:45 dist-test-slave-7g3l krb5kdc[21249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:45 dist-test-slave-7g3l krb5kdc[21249](info): set up 2 sockets
Sep 27 13:47:45 dist-test-slave-7g3l krb5kdc[21249](info): commencing operation
krb5kdc: starting...
W20260927 13:47:46.843385 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:46 dist-test-slave-7g3l krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:47:47.074154 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38209
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:47.383661 21262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:47.383960 21262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:47.384114 21262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:47.390919 21262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:47.391081 21262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:47.391203 21262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:47.391309 21262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:47.391398 21262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:47.400467 21262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:47.401599 21262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:47.402825 21262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:47.413882 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:47.413995 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:47.413829 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:47.414817 21262 server_base.cc:1034] running on GCE node
I20260927 13:47:47.415580 21262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:47.416834 21262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:47.418054 21262 hybrid_clock.cc:648] HybridClock initialized: now 1790516867417966 us; error 75 us; skew 500 ppm
I20260927 13:47:47.418378 21262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:47 dist-test-slave-7g3l krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516867, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:47.426005 21262 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:47:47.427449 21262 webserver.cc:460] Webserver started at http://127.1.237.254:43943/ using document root <none> and password file <none>
I20260927 13:47:47.428072 21262 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:47.428210 21262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:47.428510 21262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:47.431334 21262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "e8b12bca2ec448679463eafcdc970df6"
format_stamp: "Formatted at 2026-09-27 13:47:47 on dist-test-slave-7g3l"
server_key: "d69562c30d68e3bed58f730020005b5f"
server_key_iv: "50aeb8f52ff2f1263a7319adc2050348"
server_key_version: "encryptionkey@0"
I20260927 13:47:47.432103 21262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "e8b12bca2ec448679463eafcdc970df6"
format_stamp: "Formatted at 2026-09-27 13:47:47 on dist-test-slave-7g3l"
server_key: "d69562c30d68e3bed58f730020005b5f"
server_key_iv: "50aeb8f52ff2f1263a7319adc2050348"
server_key_version: "encryptionkey@0"
I20260927 13:47:47.437587 21262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:47:47.442193 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:47.443485 21262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 13:47:47.443650 21262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "e8b12bca2ec448679463eafcdc970df6"
format_stamp: "Formatted at 2026-09-27 13:47:47 on dist-test-slave-7g3l"
server_key: "d69562c30d68e3bed58f730020005b5f"
server_key_iv: "50aeb8f52ff2f1263a7319adc2050348"
server_key_version: "encryptionkey@0"
I20260927 13:47:47.443830 21262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:47.456465 21262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:47.460258 21262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:47.460522 21262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:47.478314 21262 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:38209
I20260927 13:47:47.478339 21328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:38209 every 8 connection(s)
I20260927 13:47:47.479580 21262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:47.482990 21329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:47.483366 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21262
I20260927 13:47:47.483554 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:47.484098 1975 external_mini_cluster.cc:1428] Setting key fcbf48e92742c994ffa5592a0a2a7175
I20260927 13:47:47.494940 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: Bootstrap starting.
Sep 27 13:47:47 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:47.497931 21329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:47.499008 21329 log.cc:824] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:47.501533 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: No bootstrap required, opened a new log
I20260927 13:47:47.504276 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:47.485845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.1.237.254:38209)
0927 13:47:47.486600 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:47.486610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:47.486646 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:47:47.488190 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:47.488209 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:47.488564 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:47.488945 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:47.490339 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:47.491584 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:47.492795 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:47.493455 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:47.493658 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:47.497721 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:47.497728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:47.497740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:47.497770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:47.500606 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:47.501251 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:47.501263 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:47.501268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:47.501412 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:47.501827 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:47.501833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:47.501836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:47.502312 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:47.502599 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:47.503034 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 13:47:47.503423 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":329,"threads_started":1}
I20260927 13:47:47.508754 21329 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } }
I20260927 13:47:47.509105 21329 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:47.509208 21329 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b12bca2ec448679463eafcdc970df6, State: Initialized, Role: FOLLOWER
I20260927 13:47:47.509779 21329 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } }
I20260927 13:47:47.510028 21329 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:47.510174 21329 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:47.510329 21329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:47.512383 21329 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } }
I20260927 13:47:47.512912 21329 leader_election.cc:302] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b12bca2ec448679463eafcdc970df6; no voters:
I20260927 13:47:47.513506 21329 leader_election.cc:288] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:47.513620 21334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:47.514289 21334 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [term 1 LEADER]: Becoming Leader. State: Replica: e8b12bca2ec448679463eafcdc970df6, State: Running, Role: LEADER
I20260927 13:47:47.514739 21334 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } }
I20260927 13:47:47.515415 21329 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:47.517254 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b12bca2ec448679463eafcdc970df6. Latest consensus state: current_term: 1 leader_uuid: "e8b12bca2ec448679463eafcdc970df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } } }
I20260927 13:47:47.517639 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:47.517553 21335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b12bca2ec448679463eafcdc970df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b12bca2ec448679463eafcdc970df6" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 38209 } } }
I20260927 13:47:47.517838 21335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:47.520644 21339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:47.526248 21339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:47.534235 21339 catalog_manager.cc:1347] Generated new cluster ID: 66bb59de0dd14de192e5731847a98491
I20260927 13:47:47.534330 21339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:47.559613 21339 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:47.561419 21339 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:47.575250 21339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: Generated new TSK 0
I20260927 13:47:47.576208 21339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.237.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
Entry for principal kudu/127.1.237.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab.
I20260927 13:47:47.649526 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:47.970878 21355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:47.971230 21355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:47.971339 21355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:47.978145 21355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:47.978231 21355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:47.978330 21355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:47.987520 21355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.193.keytab
--principal=kudu/127.1.237.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:47.988634 21355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:47.989820 21355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:48.002296 21361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.002344 21360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.002383 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:48.003352 21355 server_base.cc:1034] running on GCE node
I20260927 13:47:48.004033 21355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:48.005245 21355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:48.006472 21355 hybrid_clock.cc:648] HybridClock initialized: now 1790516868006410 us; error 60 us; skew 500 ppm
I20260927 13:47:48.006728 21355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:48 dist-test-slave-7g3l krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:48.009770 21355 init.cc:377] Logged in from keytab as kudu/127.1.237.193@KRBTEST.COM (short username kudu)
I20260927 13:47:48.011258 21355 webserver.cc:460] Webserver started at http://127.1.237.193:33423/ using document root <none> and password file <none>
I20260927 13:47:48.011936 21355 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:48.012064 21355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:48.012385 21355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:48.015290 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "68630ebe7806bdbe6c4d99df51231f73"
server_key_iv: "b0609d34c737f0ac2a55bfa671fc7122"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.016090 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "68630ebe7806bdbe6c4d99df51231f73"
server_key_iv: "b0609d34c737f0ac2a55bfa671fc7122"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.021075 21355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 13:47:48.024382 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.025671 21355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 13:47:48.025826 21355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "68630ebe7806bdbe6c4d99df51231f73"
server_key_iv: "b0609d34c737f0ac2a55bfa671fc7122"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.026011 21355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:48.039074 21355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:48.042689 21355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:48.042954 21355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:48.043912 21355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:48.045092 21355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:48.045163 21355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.045310 21355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:48.045367 21355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.077167 21355 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:46423
I20260927 13:47:48.077258 21482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:46423 every 8 connection(s)
I20260927 13:47:48.078385 21355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:48.085233 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21355
I20260927 13:47:48.085421 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:48.085964 1975 external_mini_cluster.cc:1428] Setting key 42492494522c97944667b3f57b093559
Sep 27 13:47:48 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.193@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:48.099751 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:48.080797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:51909 (local address 127.1.237.254:38209)
0927 13:47:48.081286 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:48.081290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:48.082254 (+ 964us) server_negotiation.cc:407] Connection header received
0927 13:47:48.084100 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:48.084105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:48.084194 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:48.084309 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:48.086210 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.087078 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.088934 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.089666 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.089811 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:48.093270 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:48.093278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:48.093285 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:48.093327 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:48.096640 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:48.097575 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:48.097579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:48.097581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:48.097686 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:48.098145 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:48.098148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:48.098150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:48.098373 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:48.098494 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:48.099341 (+ 847us) server_negotiation.cc:299] Negotiation successful
0927 13:47:48.099505 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
I20260927 13:47:48.101233 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:48.081409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38209 (local address 127.1.237.193:51909)
0927 13:47:48.082087 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:48.082129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:48.083838 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:48.084562 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:48.084570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:48.085233 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:48.086029 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:48.086056 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.087275 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.087281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:48.088785 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:48.088797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.089873 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.089878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:48.090083 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:48.090751 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:48.090772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:48.093037 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:48.096778 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:48.096787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:48.096804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:48.097454 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:48.097790 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:48.097797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:48.097804 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:48.098043 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:48.098489 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:48.098499 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:48.098925 (+ 426us) client_negotiation.cc:769] Sending connection context
0927 13:47:48.099265 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 13:47:48.099802 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260927 13:47:48.102710 21483 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38209
I20260927 13:47:48.103008 21483 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:48.103713 21483 heartbeater.cc:511] Master 127.1.237.254:38209 requested a full tablet report, sending...
I20260927 13:47:48.105685 21294 ts_manager.cc:194] Registered new tserver with Master: 8e19d825d15f4f45bc5ab48b0b18c7e4 (127.1.237.193:46423)
I20260927 13:47:48.107079 21294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.193@KRBTEST.COM'} at 127.1.237.193:51909
WARNING: no policy specified for kudu/127.1.237.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
Entry for principal kudu/127.1.237.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab.
I20260927 13:47:48.139371 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:48.442708 21489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:48.443008 21489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:48.443190 21489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:48.450083 21489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:48.450207 21489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:48.450373 21489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:48.459192 21489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.194.keytab
--principal=kudu/127.1.237.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:48.460294 21489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:48.461516 21489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:48.474362 21497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.474458 21494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.474426 21495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:48.475131 21489 server_base.cc:1034] running on GCE node
I20260927 13:47:48.475724 21489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:48.476505 21489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:48.477743 21489 hybrid_clock.cc:648] HybridClock initialized: now 1790516868477690 us; error 35 us; skew 500 ppm
I20260927 13:47:48.478081 21489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:48 dist-test-slave-7g3l krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:48.481319 21489 init.cc:377] Logged in from keytab as kudu/127.1.237.194@KRBTEST.COM (short username kudu)
I20260927 13:47:48.492184 21489 webserver.cc:460] Webserver started at http://127.1.237.194:39829/ using document root <none> and password file <none>
I20260927 13:47:48.492787 21489 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:48.492924 21489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:48.493211 21489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:48.495970 21489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "76fde1e13b8e4dec98969e64931d3c07"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "8106bae2549ffbca05df70b185be0994"
server_key_iv: "2ab5d5b93eb2d4b793bf6251f51a1aa8"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.496701 21489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "76fde1e13b8e4dec98969e64931d3c07"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "8106bae2549ffbca05df70b185be0994"
server_key_iv: "2ab5d5b93eb2d4b793bf6251f51a1aa8"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.501317 21489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:47:48.504421 21504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.505617 21489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:47:48.505784 21489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "76fde1e13b8e4dec98969e64931d3c07"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "8106bae2549ffbca05df70b185be0994"
server_key_iv: "2ab5d5b93eb2d4b793bf6251f51a1aa8"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.505990 21489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:48.527246 21489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:48.531229 21489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:48.531512 21489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:48.532351 21489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:48.533516 21489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:48.533615 21489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.533762 21489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:48.533833 21489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.564496 21489 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:46543
I20260927 13:47:48.564569 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:46543 every 8 connection(s)
I20260927 13:47:48.565768 21489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:48.567536 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21489
I20260927 13:47:48.567737 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:48.568291 1975 external_mini_cluster.cc:1428] Setting key ab2c90c87eb5d1e02ff55a9baf9423be
Sep 27 13:47:48 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.194@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:48.586018 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:48.568393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:47091 (local address 127.1.237.254:38209)
0927 13:47:48.568557 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:48.568561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:48.569837 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 13:47:48.571845 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:48.571850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:48.571939 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:48.572047 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:48.573411 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.574108 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.575787 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.576181 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.576310 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:48.580483 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:48.580488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:48.580490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:48.580531 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:48.582919 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:48.583810 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:48.583813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:48.583815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:48.583929 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:48.584365 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:48.584368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:48.584370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:48.584616 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:48.584719 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:48.585617 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 13:47:48.585794 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 13:47:48.587347 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:48.568974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38209 (local address 127.1.237.194:47091)
0927 13:47:48.569698 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:48.569733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:48.571598 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:48.572263 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:48.572270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:48.572729 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:48.573257 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:48.573272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.574251 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.574256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:48.575645 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:48.575666 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:48.576388 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:48.576397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:48.576615 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:48.577283 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:48.577316 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:48.580289 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:48.583050 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:48.583061 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:48.583077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:48.583698 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:48.584034 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:48.584041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:48.584047 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:48.584263 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:48.584745 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:48.584755 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:48.585166 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 13:47:48.585433 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 13:47:48.585984 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":811,"threads_started":1}
I20260927 13:47:48.588608 21617 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38209
I20260927 13:47:48.588900 21617 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:48.589757 21617 heartbeater.cc:511] Master 127.1.237.254:38209 requested a full tablet report, sending...
I20260927 13:47:48.591324 21294 ts_manager.cc:194] Registered new tserver with Master: 76fde1e13b8e4dec98969e64931d3c07 (127.1.237.194:46543)
I20260927 13:47:48.592095 21294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.194@KRBTEST.COM'} at 127.1.237.194:47091
WARNING: no policy specified for kudu/127.1.237.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
Entry for principal kudu/127.1.237.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab.
I20260927 13:47:48.620167 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:48.923969 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:48.924202 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:48.924355 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:48.931109 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:48.931254 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:48.931422 21622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:48.940083 21622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.195.keytab
--principal=kudu/127.1.237.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:48.941123 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:48.942266 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:48.954265 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.954339 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:48.954298 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:48.954990 21622 server_base.cc:1034] running on GCE node
I20260927 13:47:48.955562 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:48.956388 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:48.957589 21622 hybrid_clock.cc:648] HybridClock initialized: now 1790516868957541 us; error 48 us; skew 500 ppm
I20260927 13:47:48.957864 21622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:48 dist-test-slave-7g3l krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:48.960783 21622 init.cc:377] Logged in from keytab as kudu/127.1.237.195@KRBTEST.COM (short username kudu)
I20260927 13:47:48.962096 21622 webserver.cc:460] Webserver started at http://127.1.237.195:33917/ using document root <none> and password file <none>
I20260927 13:47:48.962656 21622 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:48.962749 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:48.962991 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:48.965780 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "f34370cb7570495788e88ff15cfc0154"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "4bd84d3c0d6c40309116fae3e92285cf"
server_key_iv: "8493a5517facf802d16a4ada8b358cfe"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.966456 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "f34370cb7570495788e88ff15cfc0154"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "4bd84d3c0d6c40309116fae3e92285cf"
server_key_iv: "8493a5517facf802d16a4ada8b358cfe"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.971143 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 13:47:48.974357 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.975658 21622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:48.975811 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "f34370cb7570495788e88ff15cfc0154"
format_stamp: "Formatted at 2026-09-27 13:47:48 on dist-test-slave-7g3l"
server_key: "4bd84d3c0d6c40309116fae3e92285cf"
server_key_iv: "8493a5517facf802d16a4ada8b358cfe"
server_key_version: "encryptionkey@0"
I20260927 13:47:48.975987 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:48.989828 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:48.993746 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:48.994036 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:48.994880 21622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:48.996094 21622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:48.996160 21622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:48.996325 21622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:48.996378 21622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:49.028177 21622 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:32911
I20260927 13:47:49.028215 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:32911 every 8 connection(s)
I20260927 13:47:49.029439 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:49.038573 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21622
I20260927 13:47:49.038757 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:49.039223 1975 external_mini_cluster.cc:1428] Setting key 61f2671627466a1abb3cd0c9c308afe5
Sep 27 13:47:49 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.195@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:47:49.047340 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.032036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:32917 (local address 127.1.237.254:38209)
0927 13:47:49.032207 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.032211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.033283 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 13:47:49.034945 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.034949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.035016 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.035100 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:49.036600 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.037308 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.038713 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.039184 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.039320 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.042558 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:49.042562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:49.042565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:49.042594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:49.044891 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.045475 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.045478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.045480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.045594 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.045925 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.045928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.045930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.046158 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:49.046267 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.046889 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.047035 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:47:49.048184 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.032517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:38209 (local address 127.1.237.195:32917)
0927 13:47:49.033141 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.033174 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.034681 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.035419 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.035430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.035951 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:49.036460 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.036474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.037429 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.037433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.038578 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.038589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.039395 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.039405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.039639 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.040293 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:47:49.040315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:49.042403 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:49.045002 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:49.045008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:49.045017 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:49.045377 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:49.045680 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:47:49.045683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:47:49.045685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:47:49.045841 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:47:49.046278 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:49.046284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:49.046558 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.046767 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.047079 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":239,"threads_started":1}
I20260927 13:47:49.049445 21750 heartbeater.cc:347] Connected to a master server at 127.1.237.254:38209
I20260927 13:47:49.049729 21750 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:49.050381 21750 heartbeater.cc:511] Master 127.1.237.254:38209 requested a full tablet report, sending...
I20260927 13:47:49.051745 21294 ts_manager.cc:194] Registered new tserver with Master: f34370cb7570495788e88ff15cfc0154 (127.1.237.195:32911)
I20260927 13:47:49.052551 21294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.237.195@KRBTEST.COM'} at 127.1.237.195:32917
I20260927 13:47:49.054100 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:49.069519 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.057887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.1.237.254:38209)
0927 13:47:49.058021 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.058025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.058425 (+ 400us) server_negotiation.cc:407] Connection header received
0927 13:47:49.058554 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.058557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.058609 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.058681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:49.059957 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.060650 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.061880 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.062335 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.062444 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.064725 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:49.064730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:49.064733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:49.064761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:49.067352 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.067901 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.067904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.067906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.068031 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.068401 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.068404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.068406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.068669 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:49.068842 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.069132 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.069270 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 13:47:49.073004 21294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:49.079735 21294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:49.100008 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.090779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.1.237.195:32911)
0927 13:47:49.091565 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.091575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.091611 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:47:49.091735 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.091742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.091974 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.092175 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:49.093279 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.094980 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.097588 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.098779 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.098916 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.099147 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.099486 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.099637 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":987,"threads_started":1}
I20260927 13:47:49.100059 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.090426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:32911 (local address 127.0.0.1:35698)
0927 13:47:49.091317 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.091373 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.091546 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.092526 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.092537 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.092578 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:49.093041 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.093056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.095137 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.095146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.097376 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.097394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.098982 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.098992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.099242 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.099295 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.099425 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.099596 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":5,"thread_start_us":456,"threads_started":1}
I20260927 13:47:49.100939 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.092437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:46423 (local address 127.0.0.1:43968)
0927 13:47:49.093073 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.093098 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.093198 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.094339 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.094345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.094367 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:49.094980 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.094993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.096291 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.096298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.098073 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.098087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.099865 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.099872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.100060 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.100098 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.100383 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.100518 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":117,"thread_start_us":693,"threads_started":1}
I20260927 13:47:49.101564 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.092977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.1.237.193:46423)
0927 13:47:49.093771 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.093780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.093809 (+ 29us) server_negotiation.cc:407] Connection header received
0927 13:47:49.093905 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.093911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.094125 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.094914 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:49.095207 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.096156 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.098252 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.099639 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.100602 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.100844 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.100917 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.101268 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":513,"threads_started":1}
I20260927 13:47:49.101578 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.091673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:46543 (local address 127.0.0.1:35148)
0927 13:47:49.092388 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.092410 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.092507 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.094604 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.094610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.094639 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:49.095092 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.095105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.097137 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.097142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.099110 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.099124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.100832 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.100839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.101004 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.101034 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.101098 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.101221 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":2,"thread_start_us":474,"threads_started":1}
I20260927 13:47:49.102622 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.092173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.1.237.194:46543)
0927 13:47:49.094038 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.094046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.094068 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:47:49.094152 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.094160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.094379 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.095215 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:49.095465 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.096989 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.099288 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.100568 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.101878 (+ 1310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.102076 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.102139 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.102299 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":1598,"threads_started":1}
I20260927 13:47:49.104251 21685 tablet_service.cc:1467] Processing CreateTablet for tablet e7e1edec35044612baff5b2325d66a9a (DEFAULT_TABLE table=test-table [id=07909a4604cc4a1fa958f005228f0ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:49.104773 21418 tablet_service.cc:1467] Processing CreateTablet for tablet e7e1edec35044612baff5b2325d66a9a (DEFAULT_TABLE table=test-table [id=07909a4604cc4a1fa958f005228f0ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:49.105717 21685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e1edec35044612baff5b2325d66a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:49.105935 21418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e1edec35044612baff5b2325d66a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:49.105805 21552 tablet_service.cc:1467] Processing CreateTablet for tablet e7e1edec35044612baff5b2325d66a9a (DEFAULT_TABLE table=test-table [id=07909a4604cc4a1fa958f005228f0ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:49.107013 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e1edec35044612baff5b2325d66a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:49.109885 21483 heartbeater.cc:503] Master 127.1.237.254:38209 was elected leader, sending a full tablet report...
I20260927 13:47:49.117794 21768 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Bootstrap starting.
I20260927 13:47:49.121476 21768 tablet_bootstrap.cc:654] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:49.121765 21770 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Bootstrap starting.
I20260927 13:47:49.122874 21768 log.cc:824] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:49.123409 21769 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Bootstrap starting.
I20260927 13:47:49.125283 21770 tablet_bootstrap.cc:654] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:49.126262 21769 tablet_bootstrap.cc:654] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:49.126456 21770 log.cc:824] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:49.126688 21768 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: No bootstrap required, opened a new log
I20260927 13:47:49.126952 21768 ts_tablet_manager.cc:1397] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 13:47:49.127416 21769 log.cc:824] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:49.129748 21770 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: No bootstrap required, opened a new log
I20260927 13:47:49.130043 21770 ts_tablet_manager.cc:1397] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 13:47:49.134310 21768 raft_consensus.cc:348] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.134655 21768 raft_consensus.cc:374] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:49.134756 21768 raft_consensus.cc:729] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e19d825d15f4f45bc5ab48b0b18c7e4, State: Initialized, Role: FOLLOWER
I20260927 13:47:49.135521 21768 consensus_queue.cc:260] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.137252 21769 tablet_bootstrap.cc:492] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: No bootstrap required, opened a new log
I20260927 13:47:49.137504 21769 ts_tablet_manager.cc:1397] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 13:47:49.138115 21768 ts_tablet_manager.cc:1428] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 13:47:49.140798 21770 raft_consensus.cc:348] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.141217 21770 raft_consensus.cc:374] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:49.141312 21770 raft_consensus.cc:729] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fde1e13b8e4dec98969e64931d3c07, State: Initialized, Role: FOLLOWER
I20260927 13:47:49.141886 21770 consensus_queue.cc:260] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.142911 21617 heartbeater.cc:503] Master 127.1.237.254:38209 was elected leader, sending a full tablet report...
I20260927 13:47:49.143473 21770 ts_tablet_manager.cc:1428] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Time spent starting tablet: real 0.013s user 0.013s sys 0.003s
I20260927 13:47:49.146441 21769 raft_consensus.cc:348] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.146770 21769 raft_consensus.cc:374] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:49.146840 21769 raft_consensus.cc:729] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34370cb7570495788e88ff15cfc0154, State: Initialized, Role: FOLLOWER
I20260927 13:47:49.147318 21769 consensus_queue.cc:260] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.148187 21750 heartbeater.cc:503] Master 127.1.237.254:38209 was elected leader, sending a full tablet report...
I20260927 13:47:49.148770 21769 ts_tablet_manager.cc:1428] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260927 13:47:49.281075 21751 tablet.cc:2369] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:49.317626 21618 tablet.cc:2369] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 13:47:49.330586 21484 tablet.cc:2369] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:49.454260 21776 raft_consensus.cc:482] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:49.454523 21776 raft_consensus.cc:504] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.456274 21776 leader_election.cc:288] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76fde1e13b8e4dec98969e64931d3c07 (127.1.237.194:46543), 8e19d825d15f4f45bc5ab48b0b18c7e4 (127.1.237.193:46423)
I20260927 13:47:49.462419 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.456629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:46543 (local address 127.1.237.195:33919)
0927 13:47:49.456827 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.456848 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.456970 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.457323 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.457327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.457355 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:49.457630 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.457645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.458941 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.458946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.460706 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.460719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.461910 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.461914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.462041 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.462066 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.462136 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.462225 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 13:47:49.462570 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.456687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:33919 (local address 127.1.237.194:46543)
0927 13:47:49.456895 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.456900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.456917 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:47:49.457123 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.457134 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.457195 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.457304 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:49.457822 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.458805 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.460858 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.461770 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.461913 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.461987 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.462184 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.462311 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:47:49.463587 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.457227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:46721 (local address 127.1.237.193:46423)
0927 13:47:49.457495 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.457503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.458025 (+ 522us) server_negotiation.cc:407] Connection header received
0927 13:47:49.458113 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.458118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.458195 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.458292 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:49.458735 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.459553 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.461508 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.462656 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.462823 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.462895 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.463221 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.463344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:47:49.463615 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.457062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:46423 (local address 127.1.237.195:46721)
0927 13:47:49.457909 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:49.457922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:49.458024 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:49.458308 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:49.458311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:49.458328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:49.458597 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.458604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.459697 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.459702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.461354 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:49.461364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.462912 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.462918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:49.463040 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.463062 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 13:47:49.463187 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 13:47:49.463305 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":511,"threads_started":1}
I20260927 13:47:49.463506 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e1edec35044612baff5b2325d66a9a" candidate_uuid: "f34370cb7570495788e88ff15cfc0154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fde1e13b8e4dec98969e64931d3c07" is_pre_election: true
I20260927 13:47:49.464053 21572 raft_consensus.cc:2454] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f34370cb7570495788e88ff15cfc0154 in term 0.
I20260927 13:47:49.464612 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e1edec35044612baff5b2325d66a9a" candidate_uuid: "f34370cb7570495788e88ff15cfc0154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" is_pre_election: true
I20260927 13:47:49.464772 21638 leader_election.cc:302] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fde1e13b8e4dec98969e64931d3c07, f34370cb7570495788e88ff15cfc0154; no voters:
I20260927 13:47:49.465134 21776 raft_consensus.cc:2781] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:49.465168 21438 raft_consensus.cc:2454] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f34370cb7570495788e88ff15cfc0154 in term 0.
I20260927 13:47:49.465284 21776 raft_consensus.cc:482] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:49.465408 21776 raft_consensus.cc:3037] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:49.467797 21776 raft_consensus.cc:504] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.468353 21776 leader_election.cc:288] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [CANDIDATE]: Term 1 election: Requested vote from peers 76fde1e13b8e4dec98969e64931d3c07 (127.1.237.194:46543), 8e19d825d15f4f45bc5ab48b0b18c7e4 (127.1.237.193:46423)
I20260927 13:47:49.468791 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e1edec35044612baff5b2325d66a9a" candidate_uuid: "f34370cb7570495788e88ff15cfc0154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fde1e13b8e4dec98969e64931d3c07"
I20260927 13:47:49.468979 21572 raft_consensus.cc:3037] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:49.468851 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e1edec35044612baff5b2325d66a9a" candidate_uuid: "f34370cb7570495788e88ff15cfc0154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4"
I20260927 13:47:49.469187 21438 raft_consensus.cc:3037] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:49.471305 21572 raft_consensus.cc:2454] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f34370cb7570495788e88ff15cfc0154 in term 1.
I20260927 13:47:49.471627 21438 raft_consensus.cc:2454] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f34370cb7570495788e88ff15cfc0154 in term 1.
I20260927 13:47:49.471791 21638 leader_election.cc:302] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fde1e13b8e4dec98969e64931d3c07, f34370cb7570495788e88ff15cfc0154; no voters:
I20260927 13:47:49.472070 21776 raft_consensus.cc:2781] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:49.472466 21776 raft_consensus.cc:686] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 1 LEADER]: Becoming Leader. State: Replica: f34370cb7570495788e88ff15cfc0154, State: Running, Role: LEADER
I20260927 13:47:49.472890 21776 consensus_queue.cc:237] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } }
I20260927 13:47:49.476881 21293 catalog_manager.cc:5495] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 reported cstate change: term changed from 0 to 1, leader changed from <none> to f34370cb7570495788e88ff15cfc0154 (127.1.237.195). New cstate: current_term: 1 leader_uuid: "f34370cb7570495788e88ff15cfc0154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34370cb7570495788e88ff15cfc0154" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 32911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 } health_report { overall_health: UNKNOWN } } }
I20260927 13:47:49.550773 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.545609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.1.237.195:32911)
0927 13:47:49.545827 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.545831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.545848 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:47:49.545973 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.545978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.546049 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.546162 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:49.546475 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.547270 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.548671 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.549034 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.549120 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.549587 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:49.550140 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:49.550253 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.550395 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.550493 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 13:47:49.558348 21438 raft_consensus.cc:1261] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 1 FOLLOWER]: Refusing update from remote peer f34370cb7570495788e88ff15cfc0154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:49.558429 21572 raft_consensus.cc:1261] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 1 FOLLOWER]: Refusing update from remote peer f34370cb7570495788e88ff15cfc0154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:49.559258 21778 consensus_queue.cc:1035] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e19d825d15f4f45bc5ab48b0b18c7e4" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:49.559635 21776 consensus_queue.cc:1035] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fde1e13b8e4dec98969e64931d3c07" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 46543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:49.573477 21781 mvcc.cc:205] Tried to move back new op lower bound from 7333957097702072320 to 7333957097364254720. Current Snapshot: MvccSnapshot[applied={T|T < 7333957097702072320}]
I20260927 13:47:49.587350 21293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:49.587615 21293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:49.590615 21293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8b12bca2ec448679463eafcdc970df6: Sending DeleteTablet for 3 replicas of tablet e7e1edec35044612baff5b2325d66a9a
I20260927 13:47:49.591642 21685 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e1edec35044612baff5b2325d66a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:49 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:35698
I20260927 13:47:49.592000 21552 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e1edec35044612baff5b2325d66a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:49 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:35148
I20260927 13:47:49.592193 21418 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e1edec35044612baff5b2325d66a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:49 UTC) from {username='kudu', principal='kudu/127.1.237.254@KRBTEST.COM'} at 127.0.0.1:43968
I20260927 13:47:49.593274 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21355
I20260927 13:47:49.604161 21790 tablet_replica.cc:330] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: stopping tablet replica
I20260927 13:47:49.604784 21790 raft_consensus.cc:2229] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:49.605623 21790 raft_consensus.cc:2258] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:49.606501 21791 tablet_replica.cc:330] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: stopping tablet replica
I20260927 13:47:49.606910 21791 raft_consensus.cc:2229] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:49.607379 21792 tablet_replica.cc:330] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: stopping tablet replica
I20260927 13:47:49.607537 21791 raft_consensus.cc:2258] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:49.607728 21792 raft_consensus.cc:2229] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:49.607933 21790 ts_tablet_manager.cc:1905] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:49.608163 21792 raft_consensus.cc:2258] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:49.609622 21791 ts_tablet_manager.cc:1905] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:49.610036 21792 ts_tablet_manager.cc:1905] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:49.614741 21790 ts_tablet_manager.cc:1918] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:49.614897 21790 log.cc:1196] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/e7e1edec35044612baff5b2325d66a9a
I20260927 13:47:49.616846 21792 ts_tablet_manager.cc:1918] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:49.617036 21792 log.cc:1196] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/e7e1edec35044612baff5b2325d66a9a
I20260927 13:47:49.617352 21791 ts_tablet_manager.cc:1918] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:49.617569 21791 log.cc:1196] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/e7e1edec35044612baff5b2325d66a9a
I20260927 13:47:49.617596 21792 ts_tablet_manager.cc:1939] T e7e1edec35044612baff5b2325d66a9a P 8e19d825d15f4f45bc5ab48b0b18c7e4: Deleting consensus metadata
Sep 27 13:47:49 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.193@KRBTEST.COM
I20260927 13:47:49.618168 21791 ts_tablet_manager.cc:1939] T e7e1edec35044612baff5b2325d66a9a P 76fde1e13b8e4dec98969e64931d3c07: Deleting consensus metadata
I20260927 13:47:49.619325 21278 catalog_manager.cc:4841] TS 8e19d825d15f4f45bc5ab48b0b18c7e4 (127.1.237.193:46423): tablet e7e1edec35044612baff5b2325d66a9a (table test-table [id=07909a4604cc4a1fa958f005228f0ab8]) successfully deleted
I20260927 13:47:49.619442 21790 ts_tablet_manager.cc:1939] T e7e1edec35044612baff5b2325d66a9a P f34370cb7570495788e88ff15cfc0154: Deleting consensus metadata
I20260927 13:47:49.619876 21278 catalog_manager.cc:4841] TS 76fde1e13b8e4dec98969e64931d3c07 (127.1.237.194:46543): tablet e7e1edec35044612baff5b2325d66a9a (table test-table [id=07909a4604cc4a1fa958f005228f0ab8]) successfully deleted
I20260927 13:47:49.620913 21278 catalog_manager.cc:4841] TS f34370cb7570495788e88ff15cfc0154 (127.1.237.195:32911): tablet e7e1edec35044612baff5b2325d66a9a (table test-table [id=07909a4604cc4a1fa958f005228f0ab8]) successfully deleted
I20260927 13:47:49.626597 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.596315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.1.237.193:46423)
0927 13:47:49.596494 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.596501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.606739 (+ 10238us) server_negotiation.cc:407] Connection header received
0927 13:47:49.606914 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.606922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.607009 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.607134 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:49.608931 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.609999 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.612951 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.613594 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.613695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.619787 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:49.619802 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:49.619821 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:49.619857 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:49.623894 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.624554 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.624563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.624566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.624720 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.625081 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.625087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.625090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.625531 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:49.625822 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.625997 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.626214 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:47:49.627147 21478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:49.713788 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21355
I20260927 13:47:49.735592 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21489
Sep 27 13:47:49 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.194@KRBTEST.COM
I20260927 13:47:49.751360 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.736415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.1.237.194:46543)
0927 13:47:49.736648 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.736654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.736677 (+ 23us) server_negotiation.cc:407] Connection header received
0927 13:47:49.736773 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.736779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.736855 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.736958 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:49.738128 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.739294 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.740605 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.741214 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.741337 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.745376 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:49.745394 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:49.745415 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:49.745463 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:49.748631 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.749298 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.749306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.749309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.749436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.749808 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.749812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.749816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.750152 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:49.750426 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.750713 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.750931 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:47:49.751972 21612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:49.836709 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21489
I20260927 13:47:49.853322 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21622
Sep 27 13:47:49 dist-test-slave-7g3l krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.195@KRBTEST.COM
I20260927 13:47:49.870683 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:49.854069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.1.237.195:32911)
0927 13:47:49.854229 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:49.854233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:49.854275 (+ 42us) server_negotiation.cc:407] Connection header received
0927 13:47:49.854491 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:49.854495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:49.854583 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:49.854665 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:49.856483 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.857255 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.859048 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:49.859658 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:49.859770 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:49.864842 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:49.864854 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:49.864868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:49.864897 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:49.867992 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.868643 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.868649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.868651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.868792 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:49.869178 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:49.869187 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:49.869196 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:49.869535 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:49.869833 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:49.870104 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 13:47:49.870337 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:47:49.871387 21745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:49.987454 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21622
I20260927 13:47:50.006281 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21262
I20260927 13:47:50.007404 21324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:50.113260 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21262
2026-09-27T13:47:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T13:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:50Z Disabled control of system clock
I20260927 13:47:50.183616 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46501
--webserver_interface=127.1.237.254
--webserver_port=0
--builtin_ntp_servers=127.1.237.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:50.496160 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:50.496420 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:50.496497 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:50.503212 21801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:50.503314 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:50.503348 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:50.503396 21801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:50.503441 21801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:50.511837 21801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:46501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:50.512890 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:50.514458 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:50.525657 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:50.525846 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:50.525810 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:50.526574 21801 server_base.cc:1034] running on GCE node
I20260927 13:47:50.527387 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:50.528664 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:50.529855 21801 hybrid_clock.cc:648] HybridClock initialized: now 1790516870529814 us; error 30 us; skew 500 ppm
I20260927 13:47:50.530195 21801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:50.532394 21801 webserver.cc:460] Webserver started at http://127.1.237.254:39975/ using document root <none> and password file <none>
I20260927 13:47:50.533022 21801 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:50.533133 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:50.533454 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:50.536234 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "79a08ac3229d44faaee2ff6d345810d1"
format_stamp: "Formatted at 2026-09-27 13:47:50 on dist-test-slave-7g3l"
server_key: "1a0ecdaa2b8f2a71801b6c8e72771f73"
server_key_iv: "dc7e1df36b52c74d248d88ffba6911ac"
server_key_version: "encryptionkey@0"
I20260927 13:47:50.537009 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "79a08ac3229d44faaee2ff6d345810d1"
format_stamp: "Formatted at 2026-09-27 13:47:50 on dist-test-slave-7g3l"
server_key: "1a0ecdaa2b8f2a71801b6c8e72771f73"
server_key_iv: "dc7e1df36b52c74d248d88ffba6911ac"
server_key_version: "encryptionkey@0"
I20260927 13:47:50.541777 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:47:50.544941 21815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:50.546115 21801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 13:47:50.546291 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "79a08ac3229d44faaee2ff6d345810d1"
format_stamp: "Formatted at 2026-09-27 13:47:50 on dist-test-slave-7g3l"
server_key: "1a0ecdaa2b8f2a71801b6c8e72771f73"
server_key_iv: "dc7e1df36b52c74d248d88ffba6911ac"
server_key_version: "encryptionkey@0"
I20260927 13:47:50.546489 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:50.563714 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:50.564452 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:50.564728 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:50.582623 21801 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:46501
I20260927 13:47:50.582659 21866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:46501 every 8 connection(s)
I20260927 13:47:50.583910 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:50.587391 21867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:50.591632 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21801
I20260927 13:47:50.591816 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:50.592264 1975 external_mini_cluster.cc:1428] Setting key 3024e78001a5005baa3146a4585d3559
I20260927 13:47:50.597501 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: Bootstrap starting.
I20260927 13:47:50.600800 21867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:50.601922 21867 log.cc:824] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:50.604943 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: No bootstrap required, opened a new log
I20260927 13:47:50.605252 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:50.593888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.1.237.254:46501)
0927 13:47:50.594855 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:50.594864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:50.594910 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:47:50.596628 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:50.596649 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:50.597041 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:50.597341 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:50.597944 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:50.599139 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:50.601019 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:50.601649 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:50.601788 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:50.602954 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:50.602964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:50.602984 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:50.603641 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:50.603709 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:50.603728 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:50.603809 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:50.604055 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 13:47:50.604365 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":364,"threads_started":1}
I20260927 13:47:50.612489 21867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } }
I20260927 13:47:50.612814 21867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:50.612936 21867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a08ac3229d44faaee2ff6d345810d1, State: Initialized, Role: FOLLOWER
I20260927 13:47:50.613459 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } }
I20260927 13:47:50.613683 21867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:50.613838 21867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:50.614012 21867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:50.616168 21867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } }
I20260927 13:47:50.616621 21867 leader_election.cc:302] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a08ac3229d44faaee2ff6d345810d1; no voters:
I20260927 13:47:50.617133 21867 leader_election.cc:288] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:50.617229 21872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:50.617923 21872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [term 1 LEADER]: Becoming Leader. State: Replica: 79a08ac3229d44faaee2ff6d345810d1, State: Running, Role: LEADER
I20260927 13:47:50.618355 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } }
I20260927 13:47:50.619266 21867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:50.621063 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a08ac3229d44faaee2ff6d345810d1. Latest consensus state: current_term: 1 leader_uuid: "79a08ac3229d44faaee2ff6d345810d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } } }
I20260927 13:47:50.621531 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:50.622040 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a08ac3229d44faaee2ff6d345810d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a08ac3229d44faaee2ff6d345810d1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 46501 } } }
I20260927 13:47:50.622370 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:50.623672 21881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:50.628558 21881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:50.636628 21881 catalog_manager.cc:1347] Generated new cluster ID: 54f60de5ed784452933c55d9485fbd13
I20260927 13:47:50.636716 21881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:50.665992 21881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:50.667279 21881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:50.673686 21881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: Generated new TSK 0
I20260927 13:47:50.674482 21881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:50.678399 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.193:0
--local_ip_for_outbound_sockets=127.1.237.193
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--builtin_ntp_servers=127.1.237.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 13:47:51.011260 21891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:51.011548 21891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:51.011633 21891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:51.018539 21891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:51.018625 21891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:51.018725 21891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.193
I20260927 13:47:51.027525 21891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.237.193
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.193
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:51.028671 21891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:51.030333 21891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:51.042833 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.042872 21897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.042940 21896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:51.043550 21891 server_base.cc:1034] running on GCE node
I20260927 13:47:51.044104 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:51.045020 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:51.046249 21891 hybrid_clock.cc:648] HybridClock initialized: now 1790516871046211 us; error 51 us; skew 500 ppm
I20260927 13:47:51.046535 21891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:51.048672 21891 webserver.cc:460] Webserver started at http://127.1.237.193:45109/ using document root <none> and password file <none>
I20260927 13:47:51.049351 21891 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:51.049467 21891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:51.049806 21891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:51.052758 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data/instance:
uuid: "51083283ad494b69a877e003c5bd38c2"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "8124051cb8ca4c5dd89a7fe68151f0b8"
server_key_iv: "c201db9bb098c6cf3179fefd9d88c661"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.053539 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance:
uuid: "51083283ad494b69a877e003c5bd38c2"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "8124051cb8ca4c5dd89a7fe68151f0b8"
server_key_iv: "c201db9bb098c6cf3179fefd9d88c661"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.058732 21891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 13:47:51.063583 21905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.065135 21891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 13:47:51.065325 21891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal
uuid: "51083283ad494b69a877e003c5bd38c2"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "8124051cb8ca4c5dd89a7fe68151f0b8"
server_key_iv: "c201db9bb098c6cf3179fefd9d88c661"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.065591 21891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:51.090657 21891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:51.091444 21891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:51.091715 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:51.092549 21891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:51.093951 21891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:51.094065 21891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.094183 21891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:51.094254 21891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.125702 21891 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.193:34707
I20260927 13:47:51.125806 22017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.193:34707 every 8 connection(s)
I20260927 13:47:51.126896 21891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/data/info.pb
I20260927 13:47:51.135128 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 21891
I20260927 13:47:51.135336 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal/instance
I20260927 13:47:51.135778 1975 external_mini_cluster.cc:1428] Setting key ab0e2f3692e06677f2b055ccab7bda92
I20260927 13:47:51.138204 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.194:0
--local_ip_for_outbound_sockets=127.1.237.194
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--builtin_ntp_servers=127.1.237.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:51.138768 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:51.129350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.193:60419 (local address 127.1.237.254:46501)
0927 13:47:51.129795 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:51.129799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:51.130716 (+ 917us) server_negotiation.cc:407] Connection header received
0927 13:47:51.132568 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:51.132574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:51.132662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:51.132774 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:51.133773 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.134506 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.136208 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.136608 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.136732 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:51.138010 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:51.138015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:51.138018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:51.138034 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:51.138070 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:51.138081 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:51.138172 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:51.138372 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 13:47:51.138458 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260927 13:47:51.139703 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:51.129759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46501 (local address 127.1.237.193:60419)
0927 13:47:51.130566 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:51.130606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:51.132296 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:51.132994 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:51.133002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:51.133083 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:51.133613 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:51.133629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.134641 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.134653 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:51.136063 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:51.136075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.136831 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.136839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:51.137068 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:51.137714 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:51.137717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:51.137755 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:51.137819 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:51.138173 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:51.138186 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:51.138195 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 13:47:51.138368 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 13:47:51.138711 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260927 13:47:51.141070 22018 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46501
I20260927 13:47:51.141355 22018 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:51.142128 22018 heartbeater.cc:511] Master 127.1.237.254:46501 requested a full tablet report, sending...
I20260927 13:47:51.144347 21832 ts_manager.cc:194] Registered new tserver with Master: 51083283ad494b69a877e003c5bd38c2 (127.1.237.193:34707)
I20260927 13:47:51.146438 21832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.193:60419
W20260927 13:47:51.453573 22022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:51.453866 22022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:51.454023 22022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:51.460798 22022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:51.460917 22022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:51.461097 22022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.194
I20260927 13:47:51.469792 22022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.237.194
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.194
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:51.471135 22022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:51.472872 22022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:51.485189 22028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.485246 22027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.485270 22030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:51.486225 22022 server_base.cc:1034] running on GCE node
I20260927 13:47:51.486843 22022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:51.487744 22022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:51.488977 22022 hybrid_clock.cc:648] HybridClock initialized: now 1790516871488923 us; error 48 us; skew 500 ppm
I20260927 13:47:51.489274 22022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:51.491496 22022 webserver.cc:460] Webserver started at http://127.1.237.194:40077/ using document root <none> and password file <none>
I20260927 13:47:51.492146 22022 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:51.492271 22022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:51.492581 22022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:51.495188 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data/instance:
uuid: "1c659bfeaad340bc999b3284300cb07e"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "c80923626f76bfc817705b64db75e56a"
server_key_iv: "d0323a24c3ec43ba0a725f307b33eceb"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.495929 22022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance:
uuid: "1c659bfeaad340bc999b3284300cb07e"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "c80923626f76bfc817705b64db75e56a"
server_key_iv: "d0323a24c3ec43ba0a725f307b33eceb"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.500440 22022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:47:51.503568 22036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.504720 22022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:47:51.504884 22022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal
uuid: "1c659bfeaad340bc999b3284300cb07e"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "c80923626f76bfc817705b64db75e56a"
server_key_iv: "d0323a24c3ec43ba0a725f307b33eceb"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.505087 22022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:51.532779 22022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:51.533556 22022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:51.533830 22022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:51.534703 22022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:51.536288 22022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:51.536406 22022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.536527 22022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:51.536620 22022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.567876 22022 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.194:42361
I20260927 13:47:51.567996 22148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.194:42361 every 8 connection(s)
I20260927 13:47:51.569167 22022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/data/info.pb
I20260927 13:47:51.572146 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22022
I20260927 13:47:51.572311 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal/instance
I20260927 13:47:51.572742 1975 external_mini_cluster.cc:1428] Setting key e2230948455c95e23d5a714ef15fcf40
I20260927 13:47:51.575489 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.237.195:0
--local_ip_for_outbound_sockets=127.1.237.195
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--builtin_ntp_servers=127.1.237.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 13:47:51.582417 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:51.571764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:54761 (local address 127.1.237.254:46501)
0927 13:47:51.571969 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:51.571975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:51.573636 (+ 1661us) server_negotiation.cc:407] Connection header received
0927 13:47:51.575596 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:51.575602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:51.575712 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:51.575818 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:51.576802 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.577927 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.579434 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.580152 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.580297 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:51.581514 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:51.581523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:51.581529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:51.581553 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:51.581596 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:51.581614 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:51.581705 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:51.581912 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 13:47:51.582002 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 13:47:51.583434 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:51.572385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46501 (local address 127.1.237.194:54761)
0927 13:47:51.573482 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:51.573524 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:51.575347 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:51.576022 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:51.576030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:51.576125 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:51.576641 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:51.576657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.578063 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.578067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:51.579273 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:51.579284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:51.580342 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:51.580346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:51.580531 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:51.581213 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:51.581215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:51.581260 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:51.581329 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:51.581705 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:51.581712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:51.581717 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 13:47:51.581880 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 13:47:51.582275 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":317,"threads_started":1}
I20260927 13:47:51.584937 22149 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46501
I20260927 13:47:51.585228 22149 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:51.585995 22149 heartbeater.cc:511] Master 127.1.237.254:46501 requested a full tablet report, sending...
I20260927 13:47:51.587494 21832 ts_manager.cc:194] Registered new tserver with Master: 1c659bfeaad340bc999b3284300cb07e (127.1.237.194:42361)
I20260927 13:47:51.588729 21832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.194:54761
W20260927 13:47:51.885125 22152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:51.885357 22152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:51.885514 22152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:51.892228 22152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:51.892328 22152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:51.892508 22152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.237.195
I20260927 13:47:51.900725 22152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.237.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.237.195
--webserver_port=0
--tserver_master_addrs=127.1.237.254:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.237.195
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:51.901813 22152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:51.903432 22152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:51.915611 22157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.915643 22158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:51.915794 22160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:51.916555 22152 server_base.cc:1034] running on GCE node
I20260927 13:47:51.917029 22152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:51.917819 22152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:51.919023 22152 hybrid_clock.cc:648] HybridClock initialized: now 1790516871918977 us; error 22 us; skew 500 ppm
I20260927 13:47:51.919380 22152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:47:51.921447 22152 webserver.cc:460] Webserver started at http://127.1.237.195:44339/ using document root <none> and password file <none>
I20260927 13:47:51.922087 22152 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:51.922194 22152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:51.922551 22152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:51.925343 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data/instance:
uuid: "17d5eb8fa4f74392a6e455553ac2cb03"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "93a5d7c2043e91719ab2e1c4a49e6b12"
server_key_iv: "8ab4c43feca313eaa38b3c4bfaa640d7"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.926007 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance:
uuid: "17d5eb8fa4f74392a6e455553ac2cb03"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "93a5d7c2043e91719ab2e1c4a49e6b12"
server_key_iv: "8ab4c43feca313eaa38b3c4bfaa640d7"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.930492 22152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 13:47:51.933660 22166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.934850 22152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:47:51.935031 22152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal
uuid: "17d5eb8fa4f74392a6e455553ac2cb03"
format_stamp: "Formatted at 2026-09-27 13:47:51 on dist-test-slave-7g3l"
server_key: "93a5d7c2043e91719ab2e1c4a49e6b12"
server_key_iv: "8ab4c43feca313eaa38b3c4bfaa640d7"
server_key_version: "encryptionkey@0"
I20260927 13:47:51.935271 22152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:51.960862 22152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:51.961562 22152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:51.961828 22152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:51.962697 22152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 13:47:51.964033 22152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 13:47:51.964143 22152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.964262 22152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 13:47:51.964345 22152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:51.996613 22152 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.195:35119
I20260927 13:47:51.996698 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.195:35119 every 8 connection(s)
I20260927 13:47:51.997936 22152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/data/info.pb
I20260927 13:47:51.999872 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22152
I20260927 13:47:52.000039 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal/instance
I20260927 13:47:52.000519 1975 external_mini_cluster.cc:1428] Setting key b98ffde82e14bb5bb098cbee8eb44138
I20260927 13:47:52.010288 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.000632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.195:47353 (local address 127.1.237.254:46501)
0927 13:47:52.000848 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.000854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.002548 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 13:47:52.004379 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.004383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.004448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.004532 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:52.005583 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.006317 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.007914 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.008358 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.008452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.009666 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:52.009670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:52.009673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:52.009689 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:52.009719 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:52.009730 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:52.009798 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.009993 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.010063 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:47:52.011325 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.001320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:46501 (local address 127.1.237.195:47353)
0927 13:47:52.002381 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.002432 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.004144 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.004773 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.004782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.004868 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:47:52.005417 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.005440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.006450 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.006454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.007766 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.007777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.008547 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.008553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.008785 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.009385 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 13:47:52.009387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:47:52.009427 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 13:47:52.009506 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:47:52.009813 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:47:52.009820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:47:52.009826 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.009969 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.010239 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":345,"threads_started":1}
I20260927 13:47:52.012645 22279 heartbeater.cc:347] Connected to a master server at 127.1.237.254:46501
I20260927 13:47:52.012934 22279 heartbeater.cc:464] Registering TS with master...
I20260927 13:47:52.013615 22279 heartbeater.cc:511] Master 127.1.237.254:46501 requested a full tablet report, sending...
I20260927 13:47:52.015228 21832 ts_manager.cc:194] Registered new tserver with Master: 17d5eb8fa4f74392a6e455553ac2cb03 (127.1.237.195:35119)
I20260927 13:47:52.015781 1975 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 13:47:52.016260 21832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.237.195:47353
I20260927 13:47:52.025960 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.020408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.1.237.254:46501)
0927 13:47:52.020630 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.020656 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.021089 (+ 433us) server_negotiation.cc:407] Connection header received
0927 13:47:52.021209 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.021213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.021281 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.021352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:52.021734 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.022428 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.023762 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.024199 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.024290 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.025277 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:52.025281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:52.025284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:52.025301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:52.025330 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:52.025342 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:52.025421 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.025531 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.025600 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:47:52.029415 21832 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 13:47:52.036033 21832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 13:47:52.056982 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.047182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:35119 (local address 127.0.0.1:49968)
0927 13:47:52.048006 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.048054 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.048281 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.049609 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.049617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.049649 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:52.050108 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.050137 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.052341 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.052346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.054313 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.054326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.056139 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.056145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.056337 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.056400 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.056514 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.056634 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":300,"threads_started":1}
I20260927 13:47:52.057695 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.047694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.1.237.194:42361)
0927 13:47:52.048464 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.048473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.049010 (+ 537us) server_negotiation.cc:407] Connection header received
0927 13:47:52.049230 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.049238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.049487 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.049656 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:52.051403 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.052934 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.055270 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.056632 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.056758 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.057015 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.057264 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.057435 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":486,"threads_started":1}
I20260927 13:47:52.058223 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.047338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.1.237.195:35119)
0927 13:47:52.048945 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.048953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.048999 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:47:52.049104 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.049112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.049368 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.050211 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:52.050511 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.052193 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.054539 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.055938 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.057401 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.057664 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.057732 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.057893 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":1271,"threads_started":1}
I20260927 13:47:52.058339 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.048010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.194:42361 (local address 127.0.0.1:55442)
0927 13:47:52.048868 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.048881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.049028 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.050039 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.050046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.050088 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:52.051220 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.051234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.053090 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.053095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.055038 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.055052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.056821 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.056827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.057034 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.057069 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.057903 (+ 834us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.058064 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":1039,"threads_started":1}
I20260927 13:47:52.058809 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.049666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.1.237.193:34707)
0927 13:47:52.050679 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.050687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.051066 (+ 379us) server_negotiation.cc:407] Connection header received
0927 13:47:52.051179 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.051208 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.051433 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.051608 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:52.052241 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.053716 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.056017 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.057321 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.057441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.057634 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.058129 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.058285 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":756,"threads_started":1}
I20260927 13:47:52.059391 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.049189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55384)
0927 13:47:52.050870 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.050901 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.051051 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.051601 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.051606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.051630 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:52.052049 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.052062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.053875 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.053880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.055787 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.055800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.057755 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.057761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.057921 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.057963 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.058065 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.058190 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"mutex_wait_us":3,"thread_start_us":1407,"threads_started":1}
I20260927 13:47:52.062001 22214 tablet_service.cc:1467] Processing CreateTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 (DEFAULT_TABLE table=test-table [id=3d34c1625bbf446bb1fb8e63d957d39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:52.062290 22084 tablet_service.cc:1467] Processing CreateTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 (DEFAULT_TABLE table=test-table [id=3d34c1625bbf446bb1fb8e63d957d39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:52.062623 21953 tablet_service.cc:1467] Processing CreateTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 (DEFAULT_TABLE table=test-table [id=3d34c1625bbf446bb1fb8e63d957d39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 13:47:52.063360 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a6e5574ba74a1e9c2c27b589fa9924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:52.063766 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a6e5574ba74a1e9c2c27b589fa9924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:52.063854 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a6e5574ba74a1e9c2c27b589fa9924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:52.074759 22297 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Bootstrap starting.
I20260927 13:47:52.077832 22298 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Bootstrap starting.
I20260927 13:47:52.078790 22299 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Bootstrap starting.
I20260927 13:47:52.079659 22297 tablet_bootstrap.cc:654] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:52.080914 22297 log.cc:824] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:52.081555 22298 tablet_bootstrap.cc:654] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:52.081950 22299 tablet_bootstrap.cc:654] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:52.082679 22298 log.cc:824] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:52.082932 22299 log.cc:824] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:52.084256 22297 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: No bootstrap required, opened a new log
I20260927 13:47:52.084497 22297 ts_tablet_manager.cc:1397] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 13:47:52.085590 22298 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: No bootstrap required, opened a new log
I20260927 13:47:52.085753 22299 tablet_bootstrap.cc:492] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: No bootstrap required, opened a new log
I20260927 13:47:52.085834 22298 ts_tablet_manager.cc:1397] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 13:47:52.085978 22299 ts_tablet_manager.cc:1397] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 13:47:52.092252 22297 raft_consensus.cc:348] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.092617 22297 raft_consensus.cc:374] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:52.092744 22297 raft_consensus.cc:729] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d5eb8fa4f74392a6e455553ac2cb03, State: Initialized, Role: FOLLOWER
I20260927 13:47:52.093353 22297 consensus_queue.cc:260] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.094453 22279 heartbeater.cc:503] Master 127.1.237.254:46501 was elected leader, sending a full tablet report...
I20260927 13:47:52.095093 22297 ts_tablet_manager.cc:1428] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 13:47:52.096259 22298 raft_consensus.cc:348] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.096259 22299 raft_consensus.cc:348] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.096676 22298 raft_consensus.cc:374] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:52.096683 22299 raft_consensus.cc:374] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:52.096782 22298 raft_consensus.cc:729] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c659bfeaad340bc999b3284300cb07e, State: Initialized, Role: FOLLOWER
I20260927 13:47:52.096778 22299 raft_consensus.cc:729] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51083283ad494b69a877e003c5bd38c2, State: Initialized, Role: FOLLOWER
I20260927 13:47:52.097357 22298 consensus_queue.cc:260] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.097358 22299 consensus_queue.cc:260] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.099637 22018 heartbeater.cc:503] Master 127.1.237.254:46501 was elected leader, sending a full tablet report...
I20260927 13:47:52.099751 22149 heartbeater.cc:503] Master 127.1.237.254:46501 was elected leader, sending a full tablet report...
I20260927 13:47:52.100069 22299 ts_tablet_manager.cc:1428] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260927 13:47:52.101543 22298 ts_tablet_manager.cc:1428] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20260927 13:47:52.104547 22305 raft_consensus.cc:482] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 13:47:52.104782 22305 raft_consensus.cc:504] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.106029 22305 leader_election.cc:288] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d5eb8fa4f74392a6e455553ac2cb03 (127.1.237.195:35119), 51083283ad494b69a877e003c5bd38c2 (127.1.237.193:34707)
I20260927 13:47:52.113005 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.106483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:49625 (local address 127.1.237.195:35119)
0927 13:47:52.106710 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.106719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.107061 (+ 342us) server_negotiation.cc:407] Connection header received
0927 13:47:52.107286 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.107293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.107378 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.107469 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:52.108290 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.109059 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.111343 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.112157 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.112268 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.112312 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.112648 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.112769 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:47:52.113080 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.106446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.195:35119 (local address 127.1.237.194:49625)
0927 13:47:52.106960 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.106977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.107121 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.107625 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.107629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.107653 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:52.108130 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.108141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.109193 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.109198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.111114 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.111127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.112296 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.112302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.112466 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.112516 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.112609 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.112715 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":51,"spinlock_wait_cycles":17664,"thread_start_us":280,"threads_started":1}
I20260927 13:47:52.113277 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.107387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.237.194:51787 (local address 127.1.237.193:34707)
0927 13:47:52.107568 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.107576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.107768 (+ 192us) server_negotiation.cc:407] Connection header received
0927 13:47:52.107942 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.107945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.108006 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.108077 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 13:47:52.108626 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.109402 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.111600 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.112368 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.112523 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.112597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.112912 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.113041 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:52.113440 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.107108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.1.237.194:51787)
0927 13:47:52.107672 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.107690 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 13:47:52.107822 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:47:52.108107 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:47:52.108117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:47:52.108139 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 13:47:52.108464 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.108480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.109528 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.109532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.111401 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:47:52.111426 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.112558 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.112562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:47:52.112748 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.112777 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 13:47:52.112863 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 13:47:52.112999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":23,"thread_start_us":286,"threads_started":1}
I20260927 13:47:52.114305 22234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a6e5574ba74a1e9c2c27b589fa9924" candidate_uuid: "1c659bfeaad340bc999b3284300cb07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" is_pre_election: true
I20260927 13:47:52.114488 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a6e5574ba74a1e9c2c27b589fa9924" candidate_uuid: "1c659bfeaad340bc999b3284300cb07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51083283ad494b69a877e003c5bd38c2" is_pre_election: true
I20260927 13:47:52.114912 22234 raft_consensus.cc:2454] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c659bfeaad340bc999b3284300cb07e in term 0.
I20260927 13:47:52.114964 21973 raft_consensus.cc:2454] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c659bfeaad340bc999b3284300cb07e in term 0.
I20260927 13:47:52.115592 22038 leader_election.cc:302] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c659bfeaad340bc999b3284300cb07e, 51083283ad494b69a877e003c5bd38c2; no voters:
I20260927 13:47:52.115923 22305 raft_consensus.cc:2781] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 13:47:52.116096 22305 raft_consensus.cc:482] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 13:47:52.116226 22305 raft_consensus.cc:3037] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:52.118462 22305 raft_consensus.cc:504] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
I20260927 13:47:52.119020 22305 leader_election.cc:288] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [CANDIDATE]: Term 1 election: Requested vote from peers 17d5eb8fa4f74392a6e455553ac2cb03 (127.1.237.195:35119), 51083283ad494b69a877e003c5bd38c2 (127.1.237.193:34707)
I20260927 13:47:52.119491 22234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a6e5574ba74a1e9c2c27b589fa9924" candidate_uuid: "1c659bfeaad340bc999b3284300cb07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d5eb8fa4f74392a6e455553ac2cb03"
I20260927 13:47:52.119685 22234 raft_consensus.cc:3037] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:52.119755 21973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a6e5574ba74a1e9c2c27b589fa9924" candidate_uuid: "1c659bfeaad340bc999b3284300cb07e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51083283ad494b69a877e003c5bd38c2"
I20260927 13:47:52.120033 21973 raft_consensus.cc:3037] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:52.122061 22234 raft_consensus.cc:2454] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c659bfeaad340bc999b3284300cb07e in term 1.
I20260927 13:47:52.122335 21973 raft_consensus.cc:2454] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c659bfeaad340bc999b3284300cb07e in term 1.
I20260927 13:47:52.122460 22038 leader_election.cc:302] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d5eb8fa4f74392a6e455553ac2cb03, 1c659bfeaad340bc999b3284300cb07e; no voters:
I20260927 13:47:52.122912 22305 raft_consensus.cc:2781] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:52.123384 22305 raft_consensus.cc:686] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 1 LEADER]: Becoming Leader. State: Replica: 1c659bfeaad340bc999b3284300cb07e, State: Running, Role: LEADER
I20260927 13:47:52.123898 22305 consensus_queue.cc:237] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } }
W20260927 13:47:52.128993 22019 tablet.cc:2369] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:47:52.128649 21831 catalog_manager.cc:5495] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c659bfeaad340bc999b3284300cb07e (127.1.237.194). New cstate: current_term: 1 leader_uuid: "1c659bfeaad340bc999b3284300cb07e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c659bfeaad340bc999b3284300cb07e" member_type: VOTER last_known_addr { host: "127.1.237.194" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260927 13:47:52.159027 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.153956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.1.237.194:42361)
0927 13:47:52.154191 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.154196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.154212 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:47:52.154421 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.154424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.154508 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.154628 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:47:52.154974 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.155731 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.157105 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.157507 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.157598 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.158006 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:47:52.158485 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:47:52.158600 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.158674 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.158728 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 13:47:52.167099 21973 raft_consensus.cc:1261] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 1 FOLLOWER]: Refusing update from remote peer 1c659bfeaad340bc999b3284300cb07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:52.167098 22234 raft_consensus.cc:1261] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 1 FOLLOWER]: Refusing update from remote peer 1c659bfeaad340bc999b3284300cb07e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:47:52.168114 22308 consensus_queue.cc:1035] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51083283ad494b69a877e003c5bd38c2" member_type: VOTER last_known_addr { host: "127.1.237.193" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:52.168449 22305 consensus_queue.cc:1035] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d5eb8fa4f74392a6e455553ac2cb03" member_type: VOTER last_known_addr { host: "127.1.237.195" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:47:52.177358 22311 mvcc.cc:205] Tried to move back new op lower bound from 7333957108386000896 to 7333957108223807488. Current Snapshot: MvccSnapshot[applied={T|T < 7333957108386000896}]
I20260927 13:47:52.194577 21831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:52.194870 21831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 13:47:52.198115 21831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a08ac3229d44faaee2ff6d345810d1: Sending DeleteTablet for 3 replicas of tablet c2a6e5574ba74a1e9c2c27b589fa9924
I20260927 13:47:52.199101 22214 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:52 UTC) from {username='slave'} at 127.0.0.1:49968
I20260927 13:47:52.199373 22084 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:52 UTC) from {username='slave'} at 127.0.0.1:55442
I20260927 13:47:52.200042 21953 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a6e5574ba74a1e9c2c27b589fa9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 13:47:52 UTC) from {username='slave'} at 127.0.0.1:55384
I20260927 13:47:52.200428 22321 tablet_replica.cc:330] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: stopping tablet replica
I20260927 13:47:52.200626 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21891
I20260927 13:47:52.201071 22321 raft_consensus.cc:2229] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:52.201718 22321 raft_consensus.cc:2258] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:52.204475 22321 ts_tablet_manager.cc:1905] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:52.213171 22323 tablet_replica.cc:330] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: stopping tablet replica
I20260927 13:47:52.213603 22323 raft_consensus.cc:2229] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 13:47:52.214124 22323 raft_consensus.cc:2258] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:52.214128 22321 ts_tablet_manager.cc:1918] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:52.214442 22321 log.cc:1196] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-2/wal/wals/c2a6e5574ba74a1e9c2c27b589fa9924
I20260927 13:47:52.215138 22321 ts_tablet_manager.cc:1939] T c2a6e5574ba74a1e9c2c27b589fa9924 P 17d5eb8fa4f74392a6e455553ac2cb03: Deleting consensus metadata
I20260927 13:47:52.217463 21817 catalog_manager.cc:4841] TS 17d5eb8fa4f74392a6e455553ac2cb03 (127.1.237.195:35119): tablet c2a6e5574ba74a1e9c2c27b589fa9924 (table test-table [id=3d34c1625bbf446bb1fb8e63d957d39a]) successfully deleted
I20260927 13:47:52.218523 22322 tablet_replica.cc:330] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: stopping tablet replica
I20260927 13:47:52.218976 22322 raft_consensus.cc:2229] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 1 LEADER]: Raft consensus shutting down.
I20260927 13:47:52.219830 22323 ts_tablet_manager.cc:1905] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:52.219846 22322 raft_consensus.cc:2258] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 13:47:52.221987 22322 ts_tablet_manager.cc:1905] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 13:47:52.226167 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.203167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.1.237.193:34707)
0927 13:47:52.203393 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.203401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.217294 (+ 13893us) server_negotiation.cc:407] Connection header received
0927 13:47:52.217747 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.217753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.217909 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.218021 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:52.219114 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.220362 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.222238 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.222835 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.222927 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.224369 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:52.224382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:52.224398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:52.225215 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:52.225283 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:52.225315 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:52.225415 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.225659 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.225770 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 13:47:52.227190 22013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:52.227285 22323 ts_tablet_manager.cc:1918] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:52.227458 22323 log.cc:1196] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-0/wal/wals/c2a6e5574ba74a1e9c2c27b589fa9924
I20260927 13:47:52.229140 22322 ts_tablet_manager.cc:1918] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 13:47:52.229313 22322 log.cc:1196] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Deleting WAL directory at /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790516567150638-1975-0/minicluster-data/ts-1/wal/wals/c2a6e5574ba74a1e9c2c27b589fa9924
I20260927 13:47:52.229871 22322 ts_tablet_manager.cc:1939] T c2a6e5574ba74a1e9c2c27b589fa9924 P 1c659bfeaad340bc999b3284300cb07e: Deleting consensus metadata
I20260927 13:47:52.232223 21816 catalog_manager.cc:4841] TS 1c659bfeaad340bc999b3284300cb07e (127.1.237.194:42361): tablet c2a6e5574ba74a1e9c2c27b589fa9924 (table test-table [id=3d34c1625bbf446bb1fb8e63d957d39a]) successfully deleted
I20260927 13:47:52.316787 22323 ts_tablet_manager.cc:1939] T c2a6e5574ba74a1e9c2c27b589fa9924 P 51083283ad494b69a877e003c5bd38c2: Deleting consensus metadata
I20260927 13:47:52.316952 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21891
I20260927 13:47:52.331383 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22022
W20260927 13:47:52.331406 21817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.237.193:34707 (error 108)
I20260927 13:47:52.333420 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.332888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55400)
0927 13:47:52.333141 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.333255 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260927 13:47:52.333706 21817 catalog_manager.cc:4584] TS 51083283ad494b69a877e003c5bd38c2 (127.1.237.193:34707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2a6e5574ba74a1e9c2c27b589fa9924: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
I20260927 13:47:52.339037 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.332657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.1.237.194:42361)
0927 13:47:52.332854 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.332860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.332995 (+ 135us) server_negotiation.cc:407] Connection header received
0927 13:47:52.333186 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.333192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.333314 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.333433 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:52.333898 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.334633 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.335994 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.336426 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.336561 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.337594 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:52.337602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:52.337615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:52.338175 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:52.338221 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:52.338244 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:52.338364 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.338574 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.338661 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 13:47:52.339749 22144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:52.387308 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.386787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55408)
0927 13:47:52.387004 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.387103 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 13:47:52.388307 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.387944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55422)
0927 13:47:52.388088 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.388165 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 13:47:52.447049 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22022
I20260927 13:47:52.448683 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.448229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55426)
0927 13:47:52.448442 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.448533 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 13:47:52.449776 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.449419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55440)
0927 13:47:52.449569 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.449646 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 13:47:52.465633 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22152
I20260927 13:47:52.480764 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.466350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.1.237.195:35119)
0927 13:47:52.466532 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:52.466539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:52.466622 (+ 83us) server_negotiation.cc:407] Connection header received
0927 13:47:52.466835 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:52.466840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:52.466918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:52.467036 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:52.467439 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.468155 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.469395 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:52.469795 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:52.469921 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:52.479445 (+ 9524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:52.479454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:52.479468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:52.480049 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:52.480095 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:52.480113 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:52.480238 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:52.480366 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 13:47:52.480470 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:47:52.481371 22274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:52.530328 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.529821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55450)
0927 13:47:52.530057 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.530148 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 13:47:52.531500 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:52.531087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.193:34707 (local address 127.0.0.1:55460)
0927 13:47:52.531318 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 13:47:52.531394 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.193:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 13:47:52.569238 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22152
I20260927 13:47:52.585973 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 21801
I20260927 13:47:52.587086 21862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:52.693614 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 21801
2026-09-27T13:47:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2582 ms)
[----------] 6 tests from EncryptionPolicyTest (20848 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:52 dist-test-slave-7g3l krb5kdc[22332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:52 dist-test-slave-7g3l krb5kdc[22332](info): set up 2 sockets
Sep 27 13:47:52 dist-test-slave-7g3l krb5kdc[22332](info): commencing operation
krb5kdc: starting...
W20260927 13:47:53.779867 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:53 dist-test-slave-7g3l krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:47:54.018651 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:54.338330 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:54.338668 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:54.338826 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:54.345996 22345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:54.346163 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:54.346275 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:54.346381 22345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:54.346472 22345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:54.355957 22345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:54.357260 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:54.358515 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:54.369798 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:54.369706 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:54.369767 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:54.370586 22345 server_base.cc:1034] running on GCE node
I20260927 13:47:54.371472 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:54.372839 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:54.374074 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790516874374045 us; error 55 us; skew 500 ppm
I20260927 13:47:54.374436 22345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:54 dist-test-slave-7g3l krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:54.377599 22345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:47:54.379063 22345 webserver.cc:460] Webserver started at http://127.0.0.1:38167/ using document root <none> and password file <none>
I20260927 13:47:54.379736 22345 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:54.379876 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:54.380167 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:54.382879 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "b007569d4d13434291ac0b14f19a36a2"
format_stamp: "Formatted at 2026-09-27 13:47:54 on dist-test-slave-7g3l"
server_key: "0c693947888dad13098f9935a484a599"
server_key_iv: "2cc0bce90ceb59fb21e8436e735def0a"
server_key_version: "encryptionkey@0"
I20260927 13:47:54.383662 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "b007569d4d13434291ac0b14f19a36a2"
format_stamp: "Formatted at 2026-09-27 13:47:54 on dist-test-slave-7g3l"
server_key: "0c693947888dad13098f9935a484a599"
server_key_iv: "2cc0bce90ceb59fb21e8436e735def0a"
server_key_version: "encryptionkey@0"
I20260927 13:47:54.388438 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:47:54.391646 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:54.392853 22345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:54.393021 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "b007569d4d13434291ac0b14f19a36a2"
format_stamp: "Formatted at 2026-09-27 13:47:54 on dist-test-slave-7g3l"
server_key: "0c693947888dad13098f9935a484a599"
server_key_iv: "2cc0bce90ceb59fb21e8436e735def0a"
server_key_version: "encryptionkey@0"
I20260927 13:47:54.393184 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:54.412380 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:54.415999 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:54.416280 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:54.433470 22345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39503
I20260927 13:47:54.433499 22411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39503 every 8 connection(s)
I20260927 13:47:54.434737 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:54.438426 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:54.440119 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22345
I20260927 13:47:54.444391 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:54.444975 1975 external_mini_cluster.cc:1428] Setting key 2643136da2a7873923a5b31f8eae8fb3
I20260927 13:47:54.448570 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2: Bootstrap starting.
I20260927 13:47:54.451963 22412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:54.453013 22412 log.cc:824] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:54.455535 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2: No bootstrap required, opened a new log
Sep 27 13:47:54 dist-test-slave-7g3l krb5kdc[22332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:47:54.462884 22412 raft_consensus.cc:348] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260927 13:47:54.463253 22412 raft_consensus.cc:374] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:54.463374 22412 raft_consensus.cc:729] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b007569d4d13434291ac0b14f19a36a2, State: Initialized, Role: FOLLOWER
I20260927 13:47:54.463985 22412 consensus_queue.cc:260] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260927 13:47:54.464296 22412 raft_consensus.cc:388] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:54.464429 22412 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:54.464608 22412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:54.466914 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:54.447080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52121 (local address 127.0.0.1:39503)
0927 13:47:54.448085 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:54.448095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:54.448134 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:47:54.449751 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:54.449772 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:54.450144 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:54.450476 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:54.452262 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:54.453529 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:54.454756 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:54.455453 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:54.455645 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:54.459448 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:54.459455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:54.459473 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:54.459505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:54.462378 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:54.463106 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:54.463112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:54.463117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:54.463338 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:54.463836 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:54.463840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:54.463844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:54.464534 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:54.464870 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:54.465299 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 13:47:54.465640 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":524,"threads_started":1}
I20260927 13:47:54.467885 22412 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260927 13:47:54.468374 22412 leader_election.cc:302] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b007569d4d13434291ac0b14f19a36a2; no voters:
I20260927 13:47:54.469029 22412 leader_election.cc:288] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:54.469141 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:54.470352 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [term 1 LEADER]: Becoming Leader. State: Replica: b007569d4d13434291ac0b14f19a36a2, State: Running, Role: LEADER
I20260927 13:47:54.470841 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:47:54.470938 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260927 13:47:54.471371 22412 sys_catalog.cc:564] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:54.473651 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b007569d4d13434291ac0b14f19a36a2. Latest consensus state: current_term: 1 leader_uuid: "b007569d4d13434291ac0b14f19a36a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } } }
I20260927 13:47:54.474064 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:54.474686 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b007569d4d13434291ac0b14f19a36a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007569d4d13434291ac0b14f19a36a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } } }
I20260927 13:47:54.475029 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:54.492161 22423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:54.497727 22423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:54.500516 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:54.485160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57533 (local address 127.0.0.1:39503)
0927 13:47:54.485320 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:54.485323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:54.485857 (+ 534us) server_negotiation.cc:407] Connection header received
0927 13:47:54.486033 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:54.486038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:54.486107 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:54.486199 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:54.487542 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:54.488263 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:54.489406 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:54.489852 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:54.489978 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:54.492769 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:54.492775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:54.492779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:47:54.492815 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:54.496793 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:54.497928 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:54.497934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:54.497939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:54.498074 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:47:54.498657 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:47:54.498663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:47:54.498667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:47:54.499071 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:54.499320 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:54.499980 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 13:47:54.500153 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 13:47:54.513402 22423 catalog_manager.cc:1347] Generated new cluster ID: 725be451a5724d0693c715019dd6c1ee
I20260927 13:47:54.513521 22423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:54.541581 22423 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:54.543540 22423 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:54.552265 22423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b007569d4d13434291ac0b14f19a36a2: Generated new TSK 0
I20260927 13:47:54.553206 22423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:54.570113 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22345
I20260927 13:47:54.571269 22407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:47:54.661638 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22345
W20260927 13:47:54.678722 22330 connection.cc:535] client connection to 127.0.0.1:39503 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39503): Connection reset by peer (error 104)
2026-09-27T13:47:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:54 dist-test-slave-7g3l krb5kdc[22445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:54 dist-test-slave-7g3l krb5kdc[22445](info): set up 2 sockets
Sep 27 13:47:54 dist-test-slave-7g3l krb5kdc[22445](info): commencing operation
krb5kdc: starting...
W20260927 13:47:55.739539 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:55 dist-test-slave-7g3l krb5kdc[22445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:47:55.961508 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:56.283416 22458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:56.283677 22458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:56.283764 22458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:56.290468 22458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:56.290562 22458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:56.290609 22458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:56.290666 22458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:56.290704 22458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:56.300094 22458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:56.301229 22458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:56.302431 22458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:56.313927 22463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:56.313877 22464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:56.313943 22466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:56.315447 22458 server_base.cc:1034] running on GCE node
I20260927 13:47:56.316116 22458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:56.317359 22458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:56.318548 22458 hybrid_clock.cc:648] HybridClock initialized: now 1790516876318511 us; error 27 us; skew 500 ppm
I20260927 13:47:56.318817 22458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:56 dist-test-slave-7g3l krb5kdc[22445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:56.321722 22458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:47:56.323058 22458 webserver.cc:460] Webserver started at http://127.0.0.1:39583/ using document root <none> and password file <none>
I20260927 13:47:56.323758 22458 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:56.323851 22458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:56.324090 22458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:56.326997 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "8cfd29dc267442f6ad86acec2be521ed"
format_stamp: "Formatted at 2026-09-27 13:47:56 on dist-test-slave-7g3l"
server_key: "c91462c04926d96fa7e043360ccabf0d"
server_key_iv: "df5ed5e76b3b05def8e00b401815f07d"
server_key_version: "encryptionkey@0"
I20260927 13:47:56.327773 22458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "8cfd29dc267442f6ad86acec2be521ed"
format_stamp: "Formatted at 2026-09-27 13:47:56 on dist-test-slave-7g3l"
server_key: "c91462c04926d96fa7e043360ccabf0d"
server_key_iv: "df5ed5e76b3b05def8e00b401815f07d"
server_key_version: "encryptionkey@0"
I20260927 13:47:56.332875 22458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 13:47:56.336199 22473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:56.337503 22458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:47:56.337677 22458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "8cfd29dc267442f6ad86acec2be521ed"
format_stamp: "Formatted at 2026-09-27 13:47:56 on dist-test-slave-7g3l"
server_key: "c91462c04926d96fa7e043360ccabf0d"
server_key_iv: "df5ed5e76b3b05def8e00b401815f07d"
server_key_version: "encryptionkey@0"
I20260927 13:47:56.337863 22458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:56.361873 22458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:56.365763 22458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:56.366060 22458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:56.384320 22458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41453
I20260927 13:47:56.384356 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41453 every 8 connection(s)
I20260927 13:47:56.385622 22458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:56.389163 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:56.392537 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22458
I20260927 13:47:56.392832 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:56.393234 1975 external_mini_cluster.cc:1428] Setting key e33e48ea630cf3458dca691c26e09527
I20260927 13:47:56.401135 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed: Bootstrap starting.
I20260927 13:47:56.404649 22525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:56.405771 22525 log.cc:824] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:56.406152 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:56.394843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36435 (local address 127.0.0.1:41453)
0927 13:47:56.395638 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:56.395647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:56.395688 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:47:56.397308 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:56.397328 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:56.397699 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:56.398000 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:56.398561 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:56.399967 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:56.401836 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:56.402465 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:56.402604 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:56.403888 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:56.403895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:56.403907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:56.404404 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:56.404478 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:56.404506 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:56.404619 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:56.405013 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 13:47:56.405190 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":416,"threads_started":1}
I20260927 13:47:56.408591 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed: No bootstrap required, opened a new log
I20260927 13:47:56.409929 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:47:56.418694 22525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } }
I20260927 13:47:56.419130 22525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:56.419252 22525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfd29dc267442f6ad86acec2be521ed, State: Initialized, Role: FOLLOWER
I20260927 13:47:56.419935 22525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } }
I20260927 13:47:56.420192 22525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:56.420369 22525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:56.420568 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:56.421664 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:56.415062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38739 (local address 127.0.0.1:41453)
0927 13:47:56.415260 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:56.415270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:56.415686 (+ 416us) server_negotiation.cc:407] Connection header received
0927 13:47:56.415827 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:56.415832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:56.415912 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:56.416017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:56.416472 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:56.417695 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:56.418938 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:47:56.419645 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:47:56.419786 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:47:56.420860 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:56.420864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:56.420867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:56.420882 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:56.420936 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:56.420952 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:56.421037 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:56.421274 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 13:47:56.421338 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:47:56.423106 22525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } }
I20260927 13:47:56.423681 22525 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfd29dc267442f6ad86acec2be521ed; no voters:
I20260927 13:47:56.424237 22525 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:56.424368 22535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:56.425055 22535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [term 1 LEADER]: Becoming Leader. State: Replica: 8cfd29dc267442f6ad86acec2be521ed, State: Running, Role: LEADER
I20260927 13:47:56.425591 22535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } }
I20260927 13:47:56.426261 22525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:56.428512 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfd29dc267442f6ad86acec2be521ed. Latest consensus state: current_term: 1 leader_uuid: "8cfd29dc267442f6ad86acec2be521ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } } }
I20260927 13:47:56.428875 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:56.429989 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfd29dc267442f6ad86acec2be521ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd29dc267442f6ad86acec2be521ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41453 } } }
I20260927 13:47:56.430271 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:56.430928 22541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:56.436697 22541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:56.445856 22541 catalog_manager.cc:1347] Generated new cluster ID: 3652fcfdca8a4995beaad63df40c2848
I20260927 13:47:56.445989 22541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:56.454937 22541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:56.456177 22541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:56.468116 22541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfd29dc267442f6ad86acec2be521ed: Generated new TSK 0
I20260927 13:47:56.469056 22541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:56.482631 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22458
W20260927 13:47:56.484211 22520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36435
W20260927 13:47:56.485092 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:47:56.485292 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22458
W20260927 13:47:56.501716 22443 connection.cc:535] client connection to 127.0.0.1:41453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41453): Connection reset by peer (error 104)
2026-09-27T13:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:56 dist-test-slave-7g3l krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:56 dist-test-slave-7g3l krb5kdc[22556](info): set up 2 sockets
Sep 27 13:47:56 dist-test-slave-7g3l krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260927 13:47:57.563606 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:57 dist-test-slave-7g3l krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:47:57.784536 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:58.106740 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:58.107080 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:58.107280 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:58.114244 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:58.114401 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:58.114503 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:58.114625 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:58.114702 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:58.124017 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:58.125252 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:58.126477 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:58.137754 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:58.137825 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:58.137784 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:58.138737 22569 server_base.cc:1034] running on GCE node
I20260927 13:47:58.139465 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:58.140664 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:58.141860 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790516878141818 us; error 28 us; skew 500 ppm
I20260927 13:47:58.142135 22569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:58 dist-test-slave-7g3l krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:58.152064 22569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:47:58.153366 22569 webserver.cc:460] Webserver started at http://127.0.0.1:37545/ using document root <none> and password file <none>
I20260927 13:47:58.153999 22569 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:58.154095 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:58.154320 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:58.157251 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "45ad8d9c297c426db95cb8482af290d6"
format_stamp: "Formatted at 2026-09-27 13:47:58 on dist-test-slave-7g3l"
server_key: "c913053e5876b1e877baa1c203e67b05"
server_key_iv: "cfbc11d40001f01a30a79de977c29f6f"
server_key_version: "encryptionkey@0"
I20260927 13:47:58.157943 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "45ad8d9c297c426db95cb8482af290d6"
format_stamp: "Formatted at 2026-09-27 13:47:58 on dist-test-slave-7g3l"
server_key: "c913053e5876b1e877baa1c203e67b05"
server_key_iv: "cfbc11d40001f01a30a79de977c29f6f"
server_key_version: "encryptionkey@0"
I20260927 13:47:58.162794 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 13:47:58.166018 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:58.167412 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:47:58.167555 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "45ad8d9c297c426db95cb8482af290d6"
format_stamp: "Formatted at 2026-09-27 13:47:58 on dist-test-slave-7g3l"
server_key: "c913053e5876b1e877baa1c203e67b05"
server_key_iv: "cfbc11d40001f01a30a79de977c29f6f"
server_key_version: "encryptionkey@0"
I20260927 13:47:58.167711 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:58.173033 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:58.177227 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:58.177452 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:47:58.195297 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41307
I20260927 13:47:58.195299 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41307 every 8 connection(s)
I20260927 13:47:58.196713 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:47:58.200277 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:47:58.206486 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22569
I20260927 13:47:58.206655 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:47:58.207231 1975 external_mini_cluster.cc:1428] Setting key e3392f14725c9bc25d908be829cc512f
I20260927 13:47:58.211256 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6: Bootstrap starting.
I20260927 13:47:58.214718 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6: Neither blocks nor log segments found. Creating new log.
I20260927 13:47:58.214813 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:58.209026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41609 (local address 127.0.0.1:41307)
0927 13:47:58.209825 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:58.209836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:58.209875 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:47:58.211602 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:58.211624 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:58.211948 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:58.212250 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:58.213291 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:58.213297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:58.213309 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:58.213322 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:58.213389 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:58.213408 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:58.213567 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:58.213710 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 13:47:58.213937 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":380,"threads_started":1}
I20260927 13:47:58.215862 22636 log.cc:824] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6: Log is configured to *not* fsync() on all Append() calls
I20260927 13:47:58.218489 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:47:58.218797 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6: No bootstrap required, opened a new log
I20260927 13:47:58.224889 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:47:58.222617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49293 (local address 127.0.0.1:41307)
0927 13:47:58.222781 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 13:47:58.222787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:47:58.223104 (+ 317us) server_negotiation.cc:407] Connection header received
0927 13:47:58.223277 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:47:58.223282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:47:58.223340 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:47:58.223457 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:47:58.224241 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:47:58.224246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:47:58.224251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:47:58.224267 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:47:58.224310 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:47:58.224335 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:47:58.224453 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 13:47:58.224530 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 13:47:58.224579 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 13:47:58.227396 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260927 13:47:58.227697 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:47:58.227767 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ad8d9c297c426db95cb8482af290d6, State: Initialized, Role: FOLLOWER
I20260927 13:47:58.228220 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260927 13:47:58.228421 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:47:58.228515 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:47:58.228636 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:47:58.230578 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260927 13:47:58.231139 22636 leader_election.cc:302] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ad8d9c297c426db95cb8482af290d6; no voters:
I20260927 13:47:58.231769 22636 leader_election.cc:288] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:47:58.231894 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:47:58.232537 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [term 1 LEADER]: Becoming Leader. State: Replica: 45ad8d9c297c426db95cb8482af290d6, State: Running, Role: LEADER
I20260927 13:47:58.233081 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } }
I20260927 13:47:58.233705 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:47:58.236312 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ad8d9c297c426db95cb8482af290d6. Latest consensus state: current_term: 1 leader_uuid: "45ad8d9c297c426db95cb8482af290d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } } }
I20260927 13:47:58.236747 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:58.236486 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ad8d9c297c426db95cb8482af290d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad8d9c297c426db95cb8482af290d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41307 } } }
I20260927 13:47:58.236876 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6 [sys.catalog]: This master's current role is: LEADER
I20260927 13:47:58.247691 22652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:47:58.252710 22652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:47:58.267791 22652 catalog_manager.cc:1347] Generated new cluster ID: 109955fa7b774e1a850b29882491cfc9
I20260927 13:47:58.267930 22652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:47:58.276675 22652 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:47:58.277048 22652 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:47:58.284215 22652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ad8d9c297c426db95cb8482af290d6: Generated new TSK 0
I20260927 13:47:58.284941 22652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:47:58.292104 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22569
W20260927 13:47:58.293203 22631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41609
W20260927 13:47:58.293820 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:47:58.293941 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22569
2026-09-27T13:47:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:47:58 dist-test-slave-7g3l krb5kdc[22667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:47:58 dist-test-slave-7g3l krb5kdc[22667](info): set up 2 sockets
Sep 27 13:47:58 dist-test-slave-7g3l krb5kdc[22667](info): commencing operation
krb5kdc: starting...
W20260927 13:47:59.375322 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:47:59 dist-test-slave-7g3l krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:47:59.601859 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:47:59.915459 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:47:59.915706 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:47:59.915792 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:47:59.922387 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:47:59.922497 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:47:59.922547 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:47:59.922603 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:47:59.922642 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:47:59.931996 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:47:59.933254 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:47:59.934418 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:47:59.945580 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:59.945643 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:47:59.945669 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:47:59.946345 22680 server_base.cc:1034] running on GCE node
I20260927 13:47:59.947029 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:47:59.948333 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:47:59.949554 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790516879949516 us; error 44 us; skew 500 ppm
I20260927 13:47:59.949890 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:47:59 dist-test-slave-7g3l krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:47:59.952950 22680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:47:59.954350 22680 webserver.cc:460] Webserver started at http://127.0.0.1:44997/ using document root <none> and password file <none>
I20260927 13:47:59.954991 22680 fs_manager.cc:359] Metadata directory not provided
I20260927 13:47:59.955143 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:47:59.955479 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:47:59.958194 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c"
format_stamp: "Formatted at 2026-09-27 13:47:59 on dist-test-slave-7g3l"
server_key: "a860786f7f9021e70721c6ab9a68a4f0"
server_key_iv: "ed4571a49739274b1283d14ef6480beb"
server_key_version: "encryptionkey@0"
I20260927 13:47:59.958901 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c"
format_stamp: "Formatted at 2026-09-27 13:47:59 on dist-test-slave-7g3l"
server_key: "a860786f7f9021e70721c6ab9a68a4f0"
server_key_iv: "ed4571a49739274b1283d14ef6480beb"
server_key_version: "encryptionkey@0"
I20260927 13:47:59.963557 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:47:59.966639 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:47:59.967800 22680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:47:59.967989 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c"
format_stamp: "Formatted at 2026-09-27 13:47:59 on dist-test-slave-7g3l"
server_key: "a860786f7f9021e70721c6ab9a68a4f0"
server_key_iv: "ed4571a49739274b1283d14ef6480beb"
server_key_version: "encryptionkey@0"
I20260927 13:47:59.968173 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:47:59.982864 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:47:59.986920 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:47:59.987228 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:00.005229 22680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35391
I20260927 13:48:00.005275 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35391 every 8 connection(s)
I20260927 13:48:00.006539 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:00.010157 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:00.011878 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22680
I20260927 13:48:00.012079 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:00.012488 1975 external_mini_cluster.cc:1428] Setting key 824a524555ba0bcd2d0bec81b0428eda
I20260927 13:48:00.024080 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c: Bootstrap starting.
Sep 27 13:48:00 dist-test-slave-7g3l krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:48:00.027376 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:00.028739 22747 log.cc:824] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:00.032068 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c: No bootstrap required, opened a new log
I20260927 13:48:00.036182 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:00.014163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41065 (local address 127.0.0.1:35391)
0927 13:48:00.014950 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:00.014969 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:00.015016 (+ 47us) server_negotiation.cc:407] Connection header received
0927 13:48:00.016641 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:00.016671 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:00.017161 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:00.017574 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:00.018823 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:00.020821 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:00.022038 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:00.023034 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:00.023244 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:00.027592 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:00.027604 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:00.027621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:00.027667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:00.031835 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:00.032508 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:00.032523 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:00.032534 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:00.032762 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:00.033110 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:00.033115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:00.033118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:00.033753 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:00.034085 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:00.034393 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 13:48:00.034822 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":363,"threads_started":1}
I20260927 13:48:00.040011 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:00.041607 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } }
I20260927 13:48:00.041927 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:00.042023 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb78c6d8e2c4e01945cbf0f973b5d3c, State: Initialized, Role: FOLLOWER
I20260927 13:48:00.042706 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } }
I20260927 13:48:00.042977 22747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:00.043116 22747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:00.043367 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:00.045920 22747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } }
I20260927 13:48:00.046387 22747 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb78c6d8e2c4e01945cbf0f973b5d3c; no voters:
I20260927 13:48:00.047079 22747 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:00.047210 22757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:00.048409 22757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [term 1 LEADER]: Becoming Leader. State: Replica: 1bb78c6d8e2c4e01945cbf0f973b5d3c, State: Running, Role: LEADER
I20260927 13:48:00.049065 22757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } }
I20260927 13:48:00.049456 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:00.054294 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } } }
I20260927 13:48:00.054647 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:00.060407 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:00.044256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37251 (local address 127.0.0.1:35391)
0927 13:48:00.044450 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:00.044457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:00.044802 (+ 345us) server_negotiation.cc:407] Connection header received
0927 13:48:00.044977 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:00.044982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:00.045069 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:00.045173 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:00.046622 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:00.047620 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:00.049333 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:00.049832 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:00.049976 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:00.052601 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:00.052611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:00.052618 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:00.052663 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:00.057253 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:00.057863 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:00.057868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:00.057871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:00.057995 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:00.058501 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:00.058506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:00.058510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:00.058805 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:00.059093 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:00.060014 (+ 921us) server_negotiation.cc:299] Negotiation successful
0927 13:48:00.060162 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:48:00.060505 22765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:00.063092 22759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb78c6d8e2c4e01945cbf0f973b5d3c. Latest consensus state: current_term: 1 leader_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb78c6d8e2c4e01945cbf0f973b5d3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35391 } } }
I20260927 13:48:00.063524 22759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:00.067049 22765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:00.077172 22765 catalog_manager.cc:1347] Generated new cluster ID: c99d51e6326341d698a1b32c8a7e7f26
I20260927 13:48:00.077324 22765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:00.110526 22765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:00.111883 22765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:00.121594 22765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb78c6d8e2c4e01945cbf0f973b5d3c: Generated new TSK 0
I20260927 13:48:00.122332 22765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:00.129910 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22680
I20260927 13:48:00.131064 22742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:00.223748 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22680
2026-09-27T13:48:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:00 dist-test-slave-7g3l krb5kdc[22779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:00 dist-test-slave-7g3l krb5kdc[22779](info): set up 2 sockets
Sep 27 13:48:00 dist-test-slave-7g3l krb5kdc[22779](info): commencing operation
krb5kdc: starting...
W20260927 13:48:01.291283 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:01 dist-test-slave-7g3l krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:01.510313 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:01.824445 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:01.824699 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:01.824790 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:01.831557 22792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:01.831650 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:01.831714 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:01.831773 22792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:01.831830 22792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:01.840828 22792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:01.841910 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:01.843106 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:01.854509 22798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:01.854564 22800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:01.854593 22797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:01.855399 22792 server_base.cc:1034] running on GCE node
I20260927 13:48:01.856076 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:01.857371 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:01.858572 22792 hybrid_clock.cc:648] HybridClock initialized: now 1790516881858523 us; error 50 us; skew 500 ppm
I20260927 13:48:01.858865 22792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:01 dist-test-slave-7g3l krb5kdc[22779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:01.861814 22792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:01.863094 22792 webserver.cc:460] Webserver started at http://127.0.0.1:38761/ using document root <none> and password file <none>
I20260927 13:48:01.863754 22792 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:01.863850 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:01.864136 22792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:01.866860 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "27376cd4a39444b5a121ecab9b9e99b5"
format_stamp: "Formatted at 2026-09-27 13:48:01 on dist-test-slave-7g3l"
server_key: "6895906ef3010e375e359a35d62e3d91"
server_key_iv: "d8d3cfada0a588fac0a22e8ae0f3cff7"
server_key_version: "encryptionkey@0"
I20260927 13:48:01.867601 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "27376cd4a39444b5a121ecab9b9e99b5"
format_stamp: "Formatted at 2026-09-27 13:48:01 on dist-test-slave-7g3l"
server_key: "6895906ef3010e375e359a35d62e3d91"
server_key_iv: "d8d3cfada0a588fac0a22e8ae0f3cff7"
server_key_version: "encryptionkey@0"
I20260927 13:48:01.872259 22792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 13:48:01.875312 22807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:01.876494 22792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:48:01.876667 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "27376cd4a39444b5a121ecab9b9e99b5"
format_stamp: "Formatted at 2026-09-27 13:48:01 on dist-test-slave-7g3l"
server_key: "6895906ef3010e375e359a35d62e3d91"
server_key_iv: "d8d3cfada0a588fac0a22e8ae0f3cff7"
server_key_version: "encryptionkey@0"
I20260927 13:48:01.876858 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:01.899708 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:01.903427 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:01.903692 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:01.921439 22792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40989
I20260927 13:48:01.921478 22858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40989 every 8 connection(s)
I20260927 13:48:01.922679 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:01.926038 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:01.928757 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22792
I20260927 13:48:01.928923 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:01.929471 1975 external_mini_cluster.cc:1428] Setting key 42bfba44d92b241d741fb01ffc0417bb
I20260927 13:48:01.937443 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5: Bootstrap starting.
I20260927 13:48:01.940770 22859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:01.941839 22859 log.cc:824] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:01.942209 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:01.931236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57545 (local address 127.0.0.1:40989)
0927 13:48:01.932012 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:01.932022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:01.932060 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:48:01.933630 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:01.933650 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:01.934022 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:01.934329 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:01.934893 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:01.936231 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:01.938115 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:01.938742 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:01.938882 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:01.940044 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:01.940050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:01.940062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:01.940527 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:01.940592 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:01.940615 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:01.940708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:01.941000 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 13:48:01.941273 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":379,"threads_started":1}
I20260927 13:48:01.944522 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5: No bootstrap required, opened a new log
I20260927 13:48:01.945703 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:01.954525 22859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } }
I20260927 13:48:01.954919 22859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:01.955046 22859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27376cd4a39444b5a121ecab9b9e99b5, State: Initialized, Role: FOLLOWER
I20260927 13:48:01.955686 22859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } }
I20260927 13:48:01.955960 22859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:01.956113 22859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:01.956313 22859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:01.957003 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:01.950751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58269 (local address 127.0.0.1:40989)
0927 13:48:01.950910 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:01.950914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:01.951236 (+ 322us) server_negotiation.cc:407] Connection header received
0927 13:48:01.951397 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:01.951400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:01.951456 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:01.951527 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:01.952045 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:01.952727 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:01.954547 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:01.955005 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:01.955154 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:01.956255 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:01.956261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:01.956266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:01.956288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:01.956330 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:01.956346 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:01.956460 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:01.956631 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 13:48:01.956691 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 13:48:01.958925 22859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } }
I20260927 13:48:01.959420 22859 leader_election.cc:302] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27376cd4a39444b5a121ecab9b9e99b5; no voters:
I20260927 13:48:01.959913 22859 leader_election.cc:288] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:01.960052 22869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:01.960831 22869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [term 1 LEADER]: Becoming Leader. State: Replica: 27376cd4a39444b5a121ecab9b9e99b5, State: Running, Role: LEADER
I20260927 13:48:01.961320 22869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } }
I20260927 13:48:01.961939 22859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:01.964380 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27376cd4a39444b5a121ecab9b9e99b5. Latest consensus state: current_term: 1 leader_uuid: "27376cd4a39444b5a121ecab9b9e99b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } } }
I20260927 13:48:01.964346 22870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27376cd4a39444b5a121ecab9b9e99b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27376cd4a39444b5a121ecab9b9e99b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40989 } } }
I20260927 13:48:01.964764 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:01.964797 22870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:01.966369 22877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:01.972244 22877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:01.981310 22877 catalog_manager.cc:1347] Generated new cluster ID: 4bbb545651b34f469e6c6e0bf6c65174
I20260927 13:48:01.981436 22877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:01.994341 22877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:01.995780 22877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:02.006160 22877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27376cd4a39444b5a121ecab9b9e99b5: Generated new TSK 0
I20260927 13:48:02.006994 22877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:02.017140 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22792
W20260927 13:48:02.018771 22854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57545
W20260927 13:48:02.019452 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:02.019591 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22792
2026-09-27T13:48:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1795 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:02 dist-test-slave-7g3l krb5kdc[22890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:02 dist-test-slave-7g3l krb5kdc[22890](info): set up 2 sockets
Sep 27 13:48:02 dist-test-slave-7g3l krb5kdc[22890](info): commencing operation
krb5kdc: starting...
W20260927 13:48:03.079586 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:03 dist-test-slave-7g3l krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:03.293907 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:03.622130 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:03.622535 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:03.622692 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:03.630182 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:03.630328 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:03.630441 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:03.630542 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:03.630630 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:03.640684 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:03.642047 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:03.643405 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:03.654783 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:03.654817 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:03.654762 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:03.655823 22903 server_base.cc:1034] running on GCE node
I20260927 13:48:03.656608 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:03.657843 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:03.659031 22903 hybrid_clock.cc:648] HybridClock initialized: now 1790516883658995 us; error 23 us; skew 500 ppm
I20260927 13:48:03.659373 22903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:03 dist-test-slave-7g3l krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:03.662216 22903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:03.663667 22903 webserver.cc:460] Webserver started at http://127.0.0.1:41081/ using document root <none> and password file <none>
I20260927 13:48:03.664318 22903 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:03.664461 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:03.664745 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:03.667575 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "65c9c210f9204f34b7272436029578c3"
format_stamp: "Formatted at 2026-09-27 13:48:03 on dist-test-slave-7g3l"
server_key: "e54e932f9df7cbeefab844224fdbd584"
server_key_iv: "9ddc48cac838270c0284bb8093bc7a58"
server_key_version: "encryptionkey@0"
I20260927 13:48:03.668288 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "65c9c210f9204f34b7272436029578c3"
format_stamp: "Formatted at 2026-09-27 13:48:03 on dist-test-slave-7g3l"
server_key: "e54e932f9df7cbeefab844224fdbd584"
server_key_iv: "9ddc48cac838270c0284bb8093bc7a58"
server_key_version: "encryptionkey@0"
I20260927 13:48:03.673027 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 13:48:03.676302 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:03.677497 22903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:48:03.677663 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "65c9c210f9204f34b7272436029578c3"
format_stamp: "Formatted at 2026-09-27 13:48:03 on dist-test-slave-7g3l"
server_key: "e54e932f9df7cbeefab844224fdbd584"
server_key_iv: "9ddc48cac838270c0284bb8093bc7a58"
server_key_version: "encryptionkey@0"
I20260927 13:48:03.677862 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:03.683058 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:03.686702 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:03.686975 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:03.704470 22903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34313
I20260927 13:48:03.704546 22969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34313 every 8 connection(s)
I20260927 13:48:03.705780 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:03.709198 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:03.714946 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 22903
I20260927 13:48:03.715114 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:03.715515 1975 external_mini_cluster.cc:1428] Setting key cf64b905b7dde1c4d0926e0865f1ffae
I20260927 13:48:03.719061 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3: Bootstrap starting.
I20260927 13:48:03.722523 22970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:03.722867 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:03.717234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56609 (local address 127.0.0.1:34313)
0927 13:48:03.718002 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:03.718012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:03.718050 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:48:03.719712 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:03.719734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:03.720037 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:03.720323 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:03.721396 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:03.721401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:03.721413 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:03.721434 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:03.721492 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:03.721510 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:03.721663 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:03.721800 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 13:48:03.721983 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":352,"threads_started":1}
I20260927 13:48:03.723647 22970 log.cc:824] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:03.726411 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:03.726430 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3: No bootstrap required, opened a new log
I20260927 13:48:03.732945 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:03.730425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33649 (local address 127.0.0.1:34313)
0927 13:48:03.730626 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:03.730633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:03.731106 (+ 473us) server_negotiation.cc:407] Connection header received
0927 13:48:03.731337 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:03.731352 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:03.731392 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:03.731500 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:03.732349 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:03.732354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:03.732361 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:03.732376 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:03.732418 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:03.732433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:03.732558 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:03.732645 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 13:48:03.732698 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 13:48:03.733946 22970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } }
I20260927 13:48:03.734393 22970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:03.734504 22970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c9c210f9204f34b7272436029578c3, State: Initialized, Role: FOLLOWER
I20260927 13:48:03.735036 22970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } }
I20260927 13:48:03.735312 22970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:03.735464 22970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:03.735636 22970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:03.737680 22970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } }
I20260927 13:48:03.738111 22970 leader_election.cc:302] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c9c210f9204f34b7272436029578c3; no voters:
I20260927 13:48:03.738698 22970 leader_election.cc:288] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:03.738798 22980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:03.739816 22980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [term 1 LEADER]: Becoming Leader. State: Replica: 65c9c210f9204f34b7272436029578c3, State: Running, Role: LEADER
I20260927 13:48:03.740268 22980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } }
I20260927 13:48:03.740607 22970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:03.742869 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c9c210f9204f34b7272436029578c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } } }
I20260927 13:48:03.743011 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c9c210f9204f34b7272436029578c3. Latest consensus state: current_term: 1 leader_uuid: "65c9c210f9204f34b7272436029578c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c9c210f9204f34b7272436029578c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34313 } } }
I20260927 13:48:03.743407 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:03.743479 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:03.745741 22988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:03.750329 22988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:03.758302 22988 catalog_manager.cc:1347] Generated new cluster ID: ebbc67625cd04ca1a9b16141b137be95
I20260927 13:48:03.758394 22988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:03.769968 22988 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:03.770314 22988 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:03.783144 22988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c9c210f9204f34b7272436029578c3: Generated new TSK 0
I20260927 13:48:03.783890 22988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:03.795957 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22903
W20260927 13:48:03.796855 22965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56609
W20260927 13:48:03.797479 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 22903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:03.797590 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 22903
2026-09-27T13:48:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:03 dist-test-slave-7g3l krb5kdc[23001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:03 dist-test-slave-7g3l krb5kdc[23001](info): set up 2 sockets
Sep 27 13:48:03 dist-test-slave-7g3l krb5kdc[23001](info): commencing operation
krb5kdc: starting...
W20260927 13:48:04.867542 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:05.089709 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:05.396682 23014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:05.396984 23014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:05.397159 23014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:05.403841 23014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:05.403982 23014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:05.404081 23014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:05.404183 23014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:05.404266 23014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:05.413138 23014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:05.414227 23014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:05.415365 23014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:05.426357 23020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:05.426406 23022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:05.426450 23019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:05.427263 23014 server_base.cc:1034] running on GCE node
I20260927 13:48:05.427948 23014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:05.429137 23014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:05.430330 23014 hybrid_clock.cc:648] HybridClock initialized: now 1790516885430291 us; error 38 us; skew 500 ppm
I20260927 13:48:05.430658 23014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:05.433681 23014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:05.435035 23014 webserver.cc:460] Webserver started at http://127.0.0.1:45505/ using document root <none> and password file <none>
I20260927 13:48:05.435967 23014 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:05.436115 23014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:05.436491 23014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:05.440104 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "b1f064f417434e018b9b6605e108b7b2"
format_stamp: "Formatted at 2026-09-27 13:48:05 on dist-test-slave-7g3l"
server_key: "ccf3c0c30e6788bc649cb19968fdc56e"
server_key_iv: "07c33248fb24ee53da0880e03cc338e9"
server_key_version: "encryptionkey@0"
I20260927 13:48:05.441010 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "b1f064f417434e018b9b6605e108b7b2"
format_stamp: "Formatted at 2026-09-27 13:48:05 on dist-test-slave-7g3l"
server_key: "ccf3c0c30e6788bc649cb19968fdc56e"
server_key_iv: "07c33248fb24ee53da0880e03cc338e9"
server_key_version: "encryptionkey@0"
I20260927 13:48:05.447672 23014 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 13:48:05.450907 23029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:05.452158 23014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:48:05.452343 23014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "b1f064f417434e018b9b6605e108b7b2"
format_stamp: "Formatted at 2026-09-27 13:48:05 on dist-test-slave-7g3l"
server_key: "ccf3c0c30e6788bc649cb19968fdc56e"
server_key_iv: "07c33248fb24ee53da0880e03cc338e9"
server_key_version: "encryptionkey@0"
I20260927 13:48:05.452525 23014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:05.466696 23014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:05.470454 23014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:05.470718 23014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:05.488094 23014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38809
I20260927 13:48:05.488125 23080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38809 every 8 connection(s)
I20260927 13:48:05.489327 23014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:05.492707 23081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:05.494132 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23014
I20260927 13:48:05.494328 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:05.494767 1975 external_mini_cluster.cc:1428] Setting key e6d9eae9244da2964eb69bb342d7ef44
I20260927 13:48:05.506505 23081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2: Bootstrap starting.
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:48:05.509940 23081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:05.511030 23081 log.cc:824] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:05.514284 23081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2: No bootstrap required, opened a new log
I20260927 13:48:05.518005 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:05.496465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:57337 (local address 127.0.0.1:38809)
0927 13:48:05.497239 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:05.497253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:05.497307 (+ 54us) server_negotiation.cc:407] Connection header received
0927 13:48:05.498963 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:05.498989 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:05.499517 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:05.499925 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:05.501133 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:05.503100 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:05.504928 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:05.505551 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:05.505677 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:05.510240 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:05.510247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:05.510259 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:05.510293 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:05.514022 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:05.514835 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:05.514844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:05.514849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:05.515015 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:05.515493 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:05.515499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:05.515502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:05.516009 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:05.516355 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:05.516833 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 13:48:05.517114 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":326,"threads_started":1}
I20260927 13:48:05.521793 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:05.523717 23081 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } }
I20260927 13:48:05.524091 23081 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:05.524204 23081 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f064f417434e018b9b6605e108b7b2, State: Initialized, Role: FOLLOWER
I20260927 13:48:05.524767 23081 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } }
I20260927 13:48:05.524999 23081 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:05.525156 23081 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:05.525317 23081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:05.527860 23081 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } }
I20260927 13:48:05.528302 23081 leader_election.cc:302] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f064f417434e018b9b6605e108b7b2; no voters:
I20260927 13:48:05.529013 23081 leader_election.cc:288] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:05.529124 23091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:05.529815 23091 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [term 1 LEADER]: Becoming Leader. State: Replica: b1f064f417434e018b9b6605e108b7b2, State: Running, Role: LEADER
I20260927 13:48:05.530336 23091 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } }
I20260927 13:48:05.531064 23081 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:05.532267 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f064f417434e018b9b6605e108b7b2. Latest consensus state: current_term: 1 leader_uuid: "b1f064f417434e018b9b6605e108b7b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } } }
I20260927 13:48:05.533107 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:05.537823 23093 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f064f417434e018b9b6605e108b7b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f064f417434e018b9b6605e108b7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38809 } } }
I20260927 13:48:05.538228 23093 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:05.543422 23096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:05.545459 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:05.526102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:55265 (local address 127.0.0.1:38809)
0927 13:48:05.526250 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:05.526262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:05.526747 (+ 485us) server_negotiation.cc:407] Connection header received
0927 13:48:05.526930 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:05.526936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:05.527009 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:05.527111 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:05.528486 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:05.529375 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:05.531199 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:05.531958 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:05.532100 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:05.536081 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:05.536090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:05.536098 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:05.536157 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:05.540801 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:05.542026 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:05.542034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:05.542040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:05.542194 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:05.542946 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:05.542951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:05.542954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:05.543451 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:05.544690 (+ 1239us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:05.544825 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 13:48:05.545007 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:48:05.549242 23096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:05.557111 23096 catalog_manager.cc:1347] Generated new cluster ID: 6f905925efbe48a49e9ee286588ff2c7
I20260927 13:48:05.557216 23096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:05.574402 23096 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:05.575610 23096 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:05.583527 23096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f064f417434e018b9b6605e108b7b2: Generated new TSK 0
I20260927 13:48:05.584244 23096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:05.591109 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23014
I20260927 13:48:05.592228 23076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:05.681056 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23014
W20260927 13:48:05.694612 22999 connection.cc:535] client connection to 127.0.0.1:38809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38809): Connection reset by peer (error 104)
2026-09-27T13:48:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23113](info): set up 2 sockets
Sep 27 13:48:05 dist-test-slave-7g3l krb5kdc[23113](info): commencing operation
krb5kdc: starting...
W20260927 13:48:06.755411 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:06 dist-test-slave-7g3l krb5kdc[23113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:06.991422 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:07.309849 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:07.310123 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:07.310302 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:07.317404 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:07.317507 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:07.317654 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:07.317755 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:07.317831 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:07.327020 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:07.328383 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:07.329622 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:07.341001 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:07.340924 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:07.341033 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:07.342096 23126 server_base.cc:1034] running on GCE node
I20260927 13:48:07.342831 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:07.344237 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:07.345466 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790516887345428 us; error 54 us; skew 500 ppm
I20260927 13:48:07.345805 23126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:07 dist-test-slave-7g3l krb5kdc[23113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:07.348816 23126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:07.350349 23126 webserver.cc:460] Webserver started at http://127.0.0.1:40947/ using document root <none> and password file <none>
I20260927 13:48:07.351058 23126 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:07.351229 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:07.351526 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:07.354452 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "7befa5d3184944e4a003208a6f2849f3"
format_stamp: "Formatted at 2026-09-27 13:48:07 on dist-test-slave-7g3l"
server_key: "0f2396f37d0a5be44cacacc353db07fc"
server_key_iv: "01684aa0e00848746454dfef361055b4"
server_key_version: "encryptionkey@0"
I20260927 13:48:07.355240 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "7befa5d3184944e4a003208a6f2849f3"
format_stamp: "Formatted at 2026-09-27 13:48:07 on dist-test-slave-7g3l"
server_key: "0f2396f37d0a5be44cacacc353db07fc"
server_key_iv: "01684aa0e00848746454dfef361055b4"
server_key_version: "encryptionkey@0"
I20260927 13:48:07.359972 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 13:48:07.363089 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:07.364435 23126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:48:07.364629 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "7befa5d3184944e4a003208a6f2849f3"
format_stamp: "Formatted at 2026-09-27 13:48:07 on dist-test-slave-7g3l"
server_key: "0f2396f37d0a5be44cacacc353db07fc"
server_key_iv: "01684aa0e00848746454dfef361055b4"
server_key_version: "encryptionkey@0"
I20260927 13:48:07.364809 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:07.380913 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:07.384415 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:07.384668 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:07.402308 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44297
I20260927 13:48:07.402346 23192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44297 every 8 connection(s)
I20260927 13:48:07.403609 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:07.404304 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23126
I20260927 13:48:07.404476 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:07.405000 1975 external_mini_cluster.cc:1428] Setting key 2509bcd9572071ce668686e979f12dd6
I20260927 13:48:07.407804 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:07.417359 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:07.406721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:57069 (local address 127.0.0.1:44297)
0927 13:48:07.407472 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:07.407483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:07.407518 (+ 35us) server_negotiation.cc:407] Connection header received
0927 13:48:07.409128 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:07.409148 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:07.409493 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:07.409859 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:07.410394 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:07.411770 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:07.413066 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:07.413759 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:07.413945 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:07.415286 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:07.415293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:07.415305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:07.415828 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:07.415897 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:07.415936 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:07.416097 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:07.416367 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:48:07.416536 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":318,"threads_started":1}
I20260927 13:48:07.418104 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3: Bootstrap starting.
I20260927 13:48:07.420902 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:07.421533 23193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:07.422468 23193 log.cc:824] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:07.425094 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3: No bootstrap required, opened a new log
I20260927 13:48:07.431721 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:07.425245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:32911 (local address 127.0.0.1:44297)
0927 13:48:07.425393 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:07.425398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:07.425834 (+ 436us) server_negotiation.cc:407] Connection header received
0927 13:48:07.425952 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:07.425956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:07.426024 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:07.426123 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:07.426627 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:07.427852 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:07.429120 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:07.429809 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:07.429955 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:07.430927 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:07.430935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:07.430944 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:07.430963 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:07.431021 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:07.431046 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:07.431194 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:07.431368 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 13:48:07.431418 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 13:48:07.434525 23193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } }
I20260927 13:48:07.434937 23193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:07.435055 23193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7befa5d3184944e4a003208a6f2849f3, State: Initialized, Role: FOLLOWER
I20260927 13:48:07.435736 23193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } }
I20260927 13:48:07.436000 23193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:07.436170 23193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:07.436348 23193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:07.439625 23193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } }
I20260927 13:48:07.440176 23193 leader_election.cc:302] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7befa5d3184944e4a003208a6f2849f3; no voters:
I20260927 13:48:07.440845 23193 leader_election.cc:288] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:07.440959 23203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:07.442994 23193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:07.443284 23203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7befa5d3184944e4a003208a6f2849f3, State: Running, Role: LEADER
I20260927 13:48:07.443774 23203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } }
I20260927 13:48:07.446249 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7befa5d3184944e4a003208a6f2849f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } } }
I20260927 13:48:07.446787 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:07.446656 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7befa5d3184944e4a003208a6f2849f3. Latest consensus state: current_term: 1 leader_uuid: "7befa5d3184944e4a003208a6f2849f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7befa5d3184944e4a003208a6f2849f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44297 } } }
I20260927 13:48:07.447031 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:07.451817 23213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:07.457367 23213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:07.465593 23213 catalog_manager.cc:1347] Generated new cluster ID: 84cc4ba0f6d54f29838ab47d7cc521ef
I20260927 13:48:07.465691 23213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:07.487063 23213 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:07.488757 23213 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:07.500401 23213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7befa5d3184944e4a003208a6f2849f3: Generated new TSK 0
I20260927 13:48:07.501276 23213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:07.511354 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23126
W20260927 13:48:07.512475 23188 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:57069
W20260927 13:48:07.513155 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23126: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:07.513324 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23126
W20260927 13:48:07.529949 23111 connection.cc:535] client connection to 127.0.0.1:44297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44297): Connection reset by peer (error 104)
2026-09-27T13:48:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:07 dist-test-slave-7g3l krb5kdc[23224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:07 dist-test-slave-7g3l krb5kdc[23224](info): set up 2 sockets
Sep 27 13:48:07 dist-test-slave-7g3l krb5kdc[23224](info): commencing operation
krb5kdc: starting...
W20260927 13:48:08.591709 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:08 dist-test-slave-7g3l krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:08.813684 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:09.122597 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:09.122911 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:09.123068 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:09.129954 23237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:09.130087 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:09.130185 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:09.130285 23237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:09.130385 23237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:09.139504 23237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:09.140625 23237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:09.141844 23237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:09.153110 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:09.153162 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:09.153199 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:09.153968 23237 server_base.cc:1034] running on GCE node
I20260927 13:48:09.154727 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:09.156067 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:09.157295 23237 hybrid_clock.cc:648] HybridClock initialized: now 1790516889157262 us; error 36 us; skew 500 ppm
I20260927 13:48:09.157620 23237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:09 dist-test-slave-7g3l krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:09.167745 23237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:09.169234 23237 webserver.cc:460] Webserver started at http://127.0.0.1:41535/ using document root <none> and password file <none>
I20260927 13:48:09.169843 23237 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:09.169951 23237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:09.170189 23237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:09.173090 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "3586abdf5cb541a0b0b795d6eafc80b3"
format_stamp: "Formatted at 2026-09-27 13:48:09 on dist-test-slave-7g3l"
server_key: "f3baa90e81507d324a15b50d5df1b33a"
server_key_iv: "d92311195bb7fb638383229a93dd1b86"
server_key_version: "encryptionkey@0"
I20260927 13:48:09.173768 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "3586abdf5cb541a0b0b795d6eafc80b3"
format_stamp: "Formatted at 2026-09-27 13:48:09 on dist-test-slave-7g3l"
server_key: "f3baa90e81507d324a15b50d5df1b33a"
server_key_iv: "d92311195bb7fb638383229a93dd1b86"
server_key_version: "encryptionkey@0"
I20260927 13:48:09.178475 23237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:48:09.181627 23252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:09.182811 23237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:48:09.182945 23237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "3586abdf5cb541a0b0b795d6eafc80b3"
format_stamp: "Formatted at 2026-09-27 13:48:09 on dist-test-slave-7g3l"
server_key: "f3baa90e81507d324a15b50d5df1b33a"
server_key_iv: "d92311195bb7fb638383229a93dd1b86"
server_key_version: "encryptionkey@0"
I20260927 13:48:09.183115 23237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:09.188596 23237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:09.192268 23237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:09.192502 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:09.210211 23237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37787
I20260927 13:48:09.210276 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37787 every 8 connection(s)
I20260927 13:48:09.211567 23237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:09.213128 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23237
I20260927 13:48:09.213284 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:09.213694 1975 external_mini_cluster.cc:1428] Setting key d9908324ab7a5718603f9f2777db9910
I20260927 13:48:09.215515 23304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:09.220942 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:09.215429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:35167 (local address 127.0.0.1:37787)
0927 13:48:09.216235 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:09.216246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:09.216284 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:48:09.217826 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:09.217844 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:09.218165 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:09.218478 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:09.219490 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:09.219495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:09.219507 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:09.219524 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:09.219596 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:09.219615 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:09.219771 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:09.219890 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 13:48:09.220092 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":408,"threads_started":1}
I20260927 13:48:09.224454 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:09.227207 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3: Bootstrap starting.
I20260927 13:48:09.230710 23304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:09.230926 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:09.228669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:42611 (local address 127.0.0.1:37787)
0927 13:48:09.228838 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:09.228843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:09.229103 (+ 260us) server_negotiation.cc:407] Connection header received
0927 13:48:09.229268 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:09.229271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:09.229309 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:09.229388 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:09.230232 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:09.230235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:09.230238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:09.230251 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:09.230284 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:09.230297 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:09.230392 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:09.230521 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 13:48:09.230561 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:48:09.231894 23304 log.cc:824] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:09.234212 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3: No bootstrap required, opened a new log
I20260927 13:48:09.241364 23304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } }
I20260927 13:48:09.241704 23304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:09.241806 23304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3586abdf5cb541a0b0b795d6eafc80b3, State: Initialized, Role: FOLLOWER
I20260927 13:48:09.242307 23304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } }
I20260927 13:48:09.242532 23304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:09.242677 23304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:09.242852 23304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:09.244904 23304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } }
I20260927 13:48:09.245347 23304 leader_election.cc:302] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3586abdf5cb541a0b0b795d6eafc80b3; no voters:
I20260927 13:48:09.245893 23304 leader_election.cc:288] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:09.246047 23314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:09.246708 23314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3586abdf5cb541a0b0b795d6eafc80b3, State: Running, Role: LEADER
I20260927 13:48:09.247287 23314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } }
I20260927 13:48:09.247815 23304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:09.251477 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } } }
I20260927 13:48:09.251796 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3586abdf5cb541a0b0b795d6eafc80b3. Latest consensus state: current_term: 1 leader_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3586abdf5cb541a0b0b795d6eafc80b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37787 } } }
I20260927 13:48:09.251967 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:09.252121 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:09.253320 23324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:09.258265 23324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:09.266639 23324 catalog_manager.cc:1347] Generated new cluster ID: 037e4222b7654c859af9c5a5853feba9
I20260927 13:48:09.266741 23324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:09.286468 23324 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:09.286875 23324 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:09.293865 23324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3586abdf5cb541a0b0b795d6eafc80b3: Generated new TSK 0
I20260927 13:48:09.294574 23324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:09.300143 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23237
W20260927 13:48:09.301211 23299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:35167
W20260927 13:48:09.301805 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:09.301946 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23237
2026-09-27T13:48:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1788 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:09 dist-test-slave-7g3l krb5kdc[23335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:09 dist-test-slave-7g3l krb5kdc[23335](info): set up 2 sockets
Sep 27 13:48:09 dist-test-slave-7g3l krb5kdc[23335](info): commencing operation
krb5kdc: starting...
W20260927 13:48:10.359535 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:10 dist-test-slave-7g3l krb5kdc[23335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:10.583191 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:10.892987 23348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:10.893258 23348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:10.893428 23348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:10.900401 23348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:10.900511 23348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:10.900589 23348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:10.900691 23348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:10.900744 23348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:10.909799 23348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:10.910971 23348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:10.912211 23348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:10.923559 23353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:10.923601 23354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:10.923502 23356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:10.924417 23348 server_base.cc:1034] running on GCE node
I20260927 13:48:10.925189 23348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:10.926522 23348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:10.927759 23348 hybrid_clock.cc:648] HybridClock initialized: now 1790516890927723 us; error 64 us; skew 500 ppm
I20260927 13:48:10.928110 23348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:10 dist-test-slave-7g3l krb5kdc[23335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:10.930989 23348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:10.932340 23348 webserver.cc:460] Webserver started at http://127.0.0.1:44267/ using document root <none> and password file <none>
I20260927 13:48:10.933032 23348 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:10.933172 23348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:10.933476 23348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:10.936353 23348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "075411c1dcf942a89f600c69f3a80ca1"
format_stamp: "Formatted at 2026-09-27 13:48:10 on dist-test-slave-7g3l"
server_key: "33f0ee960e9983de44af0ca5365dfc53"
server_key_iv: "70dd9bbdd30fe3936c8bbfa0c126fe1f"
server_key_version: "encryptionkey@0"
I20260927 13:48:10.937104 23348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "075411c1dcf942a89f600c69f3a80ca1"
format_stamp: "Formatted at 2026-09-27 13:48:10 on dist-test-slave-7g3l"
server_key: "33f0ee960e9983de44af0ca5365dfc53"
server_key_iv: "70dd9bbdd30fe3936c8bbfa0c126fe1f"
server_key_version: "encryptionkey@0"
I20260927 13:48:10.942099 23348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 13:48:10.945286 23363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:10.946432 23348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:48:10.946619 23348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "075411c1dcf942a89f600c69f3a80ca1"
format_stamp: "Formatted at 2026-09-27 13:48:10 on dist-test-slave-7g3l"
server_key: "33f0ee960e9983de44af0ca5365dfc53"
server_key_iv: "70dd9bbdd30fe3936c8bbfa0c126fe1f"
server_key_version: "encryptionkey@0"
I20260927 13:48:10.946820 23348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:10.959311 23348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:10.962723 23348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:10.962996 23348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:10.979846 23348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36399
I20260927 13:48:10.979873 23414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36399 every 8 connection(s)
I20260927 13:48:10.981092 23348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:10.982789 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23348
I20260927 13:48:10.982944 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:10.983489 1975 external_mini_cluster.cc:1428] Setting key 19dac4bc24b3a9f46e85268f1c77d679
I20260927 13:48:10.985219 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:48:10 dist-test-slave-7g3l krb5kdc[23335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 13:48:10.997965 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1: Bootstrap starting.
I20260927 13:48:11.000869 23415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:11.001893 23415 log.cc:824] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:11.004868 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1: No bootstrap required, opened a new log
I20260927 13:48:11.005712 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:10.985104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:56943 (local address 127.0.0.1:36399)
0927 13:48:10.985894 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:10.985908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:10.985954 (+ 46us) server_negotiation.cc:407] Connection header received
0927 13:48:10.988145 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:10.988169 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:10.988648 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:10.989047 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:10.990264 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:10.992304 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:10.993507 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:10.994470 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:10.994657 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:10.998783 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:10.998790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:10.998802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:10.998835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:11.001641 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:11.002623 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:11.002629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:11.002636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:11.002827 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:11.003250 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:11.003271 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:11.003274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:11.003715 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:11.003958 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:11.004458 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 13:48:11.004721 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260927 13:48:11.009472 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:11.014600 23415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 13:48:11.015015 23415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:11.015127 23415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075411c1dcf942a89f600c69f3a80ca1, State: Initialized, Role: FOLLOWER
I20260927 13:48:11.015797 23415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 13:48:11.016039 23415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:11.016184 23415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:11.016361 23415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:11.018447 23415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 13:48:11.018890 23415 leader_election.cc:302] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075411c1dcf942a89f600c69f3a80ca1; no voters:
I20260927 13:48:11.019565 23415 leader_election.cc:288] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:11.019693 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:11.020835 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [term 1 LEADER]: Becoming Leader. State: Replica: 075411c1dcf942a89f600c69f3a80ca1, State: Running, Role: LEADER
I20260927 13:48:11.021396 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } }
I20260927 13:48:11.021798 23415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:11.024500 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075411c1dcf942a89f600c69f3a80ca1. Latest consensus state: current_term: 1 leader_uuid: "075411c1dcf942a89f600c69f3a80ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } } }
I20260927 13:48:11.025002 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:11.025936 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "075411c1dcf942a89f600c69f3a80ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075411c1dcf942a89f600c69f3a80ca1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36399 } } }
I20260927 13:48:11.026296 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:11.031716 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:11.014478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:32869 (local address 127.0.0.1:36399)
0927 13:48:11.014612 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:11.014615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:11.014882 (+ 267us) server_negotiation.cc:407] Connection header received
0927 13:48:11.015085 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:11.015106 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:11.015230 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:11.015339 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:11.016883 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:11.017682 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:11.018932 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:11.019520 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:11.019663 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:11.022142 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:11.022149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:11.022157 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:11.022182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:11.025612 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:11.027578 (+ 1966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:11.027586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:11.027592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:11.027793 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:11.028497 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:11.028504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:11.028509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:11.028968 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:11.030586 (+ 1618us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:11.030793 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 13:48:11.031034 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 13:48:11.034225 23430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:11.042330 23430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:11.050297 23430 catalog_manager.cc:1347] Generated new cluster ID: 35ff54e1edc44a44a761dfe95a444310
I20260927 13:48:11.050391 23430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:11.059885 23430 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:11.060981 23430 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:11.068853 23430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 075411c1dcf942a89f600c69f3a80ca1: Generated new TSK 0
I20260927 13:48:11.069526 23430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:11.076594 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23348
I20260927 13:48:11.077696 23410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:11.169379 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23348
W20260927 13:48:11.185817 23333 connection.cc:535] client connection to 127.0.0.1:36399 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36399): Connection reset by peer (error 104)
2026-09-27T13:48:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:11 dist-test-slave-7g3l krb5kdc[23447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:11 dist-test-slave-7g3l krb5kdc[23447](info): set up 2 sockets
Sep 27 13:48:11 dist-test-slave-7g3l krb5kdc[23447](info): commencing operation
krb5kdc: starting...
W20260927 13:48:12.231526 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:12 dist-test-slave-7g3l krb5kdc[23447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:12.459127 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:12.766160 23460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:12.766552 23460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:12.766731 23460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:12.773627 23460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:12.773765 23460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:12.773846 23460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:12.773933 23460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:12.773996 23460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:12.782748 23460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:12.783908 23460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:12.785137 23460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:12.796213 23465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:12.796304 23466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:12.796332 23468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:12.797359 23460 server_base.cc:1034] running on GCE node
I20260927 13:48:12.798143 23460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:12.799620 23460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:12.800863 23460 hybrid_clock.cc:648] HybridClock initialized: now 1790516892800829 us; error 45 us; skew 500 ppm
I20260927 13:48:12.801234 23460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:12 dist-test-slave-7g3l krb5kdc[23447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:12.804283 23460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:12.805614 23460 webserver.cc:460] Webserver started at http://127.0.0.1:40533/ using document root <none> and password file <none>
I20260927 13:48:12.806241 23460 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:12.806375 23460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:12.806674 23460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:12.809644 23460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "64d8f6938e7c4fc9911fda4ac45f694b"
format_stamp: "Formatted at 2026-09-27 13:48:12 on dist-test-slave-7g3l"
server_key: "4b35d92690adcbb0596776b17915790a"
server_key_iv: "7e75377424bbe3823083e22ab7f89793"
server_key_version: "encryptionkey@0"
I20260927 13:48:12.810375 23460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "64d8f6938e7c4fc9911fda4ac45f694b"
format_stamp: "Formatted at 2026-09-27 13:48:12 on dist-test-slave-7g3l"
server_key: "4b35d92690adcbb0596776b17915790a"
server_key_iv: "7e75377424bbe3823083e22ab7f89793"
server_key_version: "encryptionkey@0"
I20260927 13:48:12.815348 23460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 13:48:12.818572 23475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:12.819857 23460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:48:12.820041 23460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "64d8f6938e7c4fc9911fda4ac45f694b"
format_stamp: "Formatted at 2026-09-27 13:48:12 on dist-test-slave-7g3l"
server_key: "4b35d92690adcbb0596776b17915790a"
server_key_iv: "7e75377424bbe3823083e22ab7f89793"
server_key_version: "encryptionkey@0"
I20260927 13:48:12.820225 23460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:12.843011 23460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:12.847070 23460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:12.847345 23460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:12.864512 23460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41615
I20260927 13:48:12.864547 23526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41615 every 8 connection(s)
I20260927 13:48:12.865760 23460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:12.869282 23527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:12.869645 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23460
I20260927 13:48:12.869804 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:12.870217 1975 external_mini_cluster.cc:1428] Setting key 611ff30cba87e19a734d5c9b533f5320
I20260927 13:48:12.881248 23527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b: Bootstrap starting.
I20260927 13:48:12.882191 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:12.871929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:58655 (local address 127.0.0.1:41615)
0927 13:48:12.872637 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:12.872647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:12.872683 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:48:12.874109 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:12.874127 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:12.874500 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:12.874952 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:12.875516 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:12.876704 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:12.877972 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:12.878649 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:12.878888 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:12.880058 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:12.880065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:12.880077 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:12.880556 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:12.880622 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:12.880650 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:12.880823 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:12.881099 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 13:48:12.881254 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":314,"threads_started":1}
I20260927 13:48:12.884450 23527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:12.885540 23527 log.cc:824] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:12.885819 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:12.888090 23527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b: No bootstrap required, opened a new log
I20260927 13:48:12.896123 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:12.889967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:59063 (local address 127.0.0.1:41615)
0927 13:48:12.890134 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:12.890140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:12.890565 (+ 425us) server_negotiation.cc:407] Connection header received
0927 13:48:12.890769 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:12.890780 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:12.890850 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:12.890973 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:12.891546 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:12.892280 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:12.893574 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:12.894048 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:12.894184 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:12.895352 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:12.895358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:12.895363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:12.895383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:12.895426 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:12.895447 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:12.895569 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:12.895750 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 13:48:12.895809 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:48:12.896078 23527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } }
I20260927 13:48:12.896553 23527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:12.896678 23527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d8f6938e7c4fc9911fda4ac45f694b, State: Initialized, Role: FOLLOWER
I20260927 13:48:12.897294 23527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } }
I20260927 13:48:12.897593 23527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:12.897734 23527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:12.897861 23527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:12.899933 23527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } }
I20260927 13:48:12.900345 23527 leader_election.cc:302] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d8f6938e7c4fc9911fda4ac45f694b; no voters:
I20260927 13:48:12.900830 23527 leader_election.cc:288] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:12.900978 23537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:12.901654 23537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [term 1 LEADER]: Becoming Leader. State: Replica: 64d8f6938e7c4fc9911fda4ac45f694b, State: Running, Role: LEADER
I20260927 13:48:12.902184 23537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } }
I20260927 13:48:12.902824 23527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:12.905160 23538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } } }
I20260927 13:48:12.905468 23538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:12.906179 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d8f6938e7c4fc9911fda4ac45f694b. Latest consensus state: current_term: 1 leader_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d8f6938e7c4fc9911fda4ac45f694b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41615 } } }
I20260927 13:48:12.906451 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:12.907001 23545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:12.911945 23545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:12.920535 23545 catalog_manager.cc:1347] Generated new cluster ID: 843d988696bb45288625464a7efa1e0c
I20260927 13:48:12.920645 23545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:12.940052 23545 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:12.941303 23545 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:12.956616 23545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d8f6938e7c4fc9911fda4ac45f694b: Generated new TSK 0
I20260927 13:48:12.957355 23545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:12.971431 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23460
W20260927 13:48:12.972456 23522 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:58655
W20260927 13:48:12.973155 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23460: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:12.973304 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23460
W20260927 13:48:12.990983 23445 connection.cc:535] client connection to 127.0.0.1:41615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41615): Connection reset by peer (error 104)
2026-09-27T13:48:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:13 dist-test-slave-7g3l krb5kdc[23558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:13 dist-test-slave-7g3l krb5kdc[23558](info): set up 2 sockets
Sep 27 13:48:13 dist-test-slave-7g3l krb5kdc[23558](info): commencing operation
krb5kdc: starting...
W20260927 13:48:14.063512 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:14 dist-test-slave-7g3l krb5kdc[23558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 13:48:14.280390 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:14.580960 23571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:14.581228 23571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:14.581377 23571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:14.588397 23571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:14.588518 23571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:14.588619 23571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:14.588727 23571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:14.588816 23571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:14.597793 23571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:14.598939 23571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:14.600139 23571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:14.610967 23576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:14.610980 23579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:14.610999 23577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:14.611716 23571 server_base.cc:1034] running on GCE node
I20260927 13:48:14.612458 23571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:14.613762 23571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:14.614990 23571 hybrid_clock.cc:648] HybridClock initialized: now 1790516894614955 us; error 45 us; skew 500 ppm
I20260927 13:48:14.615363 23571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:14 dist-test-slave-7g3l krb5kdc[23558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:14.621701 23571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 13:48:14.623126 23571 webserver.cc:460] Webserver started at http://127.0.0.1:34173/ using document root <none> and password file <none>
I20260927 13:48:14.623798 23571 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:14.623939 23571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:14.624214 23571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:14.627061 23571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "d1d111958b3b44e9b75ab7931cb7ac0e"
format_stamp: "Formatted at 2026-09-27 13:48:14 on dist-test-slave-7g3l"
server_key: "f1af98b895e22dabf9e3489739cb566e"
server_key_iv: "a550b1767303a9942daf55cee89ebc4b"
server_key_version: "encryptionkey@0"
I20260927 13:48:14.627902 23571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "d1d111958b3b44e9b75ab7931cb7ac0e"
format_stamp: "Formatted at 2026-09-27 13:48:14 on dist-test-slave-7g3l"
server_key: "f1af98b895e22dabf9e3489739cb566e"
server_key_iv: "a550b1767303a9942daf55cee89ebc4b"
server_key_version: "encryptionkey@0"
I20260927 13:48:14.632776 23571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:48:14.636106 23586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:14.637367 23571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:48:14.637559 23571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "d1d111958b3b44e9b75ab7931cb7ac0e"
format_stamp: "Formatted at 2026-09-27 13:48:14 on dist-test-slave-7g3l"
server_key: "f1af98b895e22dabf9e3489739cb566e"
server_key_iv: "a550b1767303a9942daf55cee89ebc4b"
server_key_version: "encryptionkey@0"
I20260927 13:48:14.637756 23571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:14.643479 23571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:14.647141 23571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:14.647442 23571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:14.665338 23571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38591
I20260927 13:48:14.665381 23637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38591 every 8 connection(s)
I20260927 13:48:14.666640 23571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:14.667805 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23571
I20260927 13:48:14.667989 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:14.668534 1975 external_mini_cluster.cc:1428] Setting key db85b292bfc80781d3c962bd13e17c44
I20260927 13:48:14.671029 23638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:14.676055 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:14.670285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:43057 (local address 127.0.0.1:38591)
0927 13:48:14.671027 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:14.671039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:14.671105 (+ 66us) server_negotiation.cc:407] Connection header received
0927 13:48:14.672711 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:14.672731 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:14.673054 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:14.673459 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:14.674479 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:14.674486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:14.674498 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:14.674510 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:14.674576 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:14.674598 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:14.674832 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:14.674967 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 13:48:14.675146 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":311,"threads_started":1}
I20260927 13:48:14.679558 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:14.682008 23638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e: Bootstrap starting.
I20260927 13:48:14.685463 23638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:14.686031 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:14.683725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:51065 (local address 127.0.0.1:38591)
0927 13:48:14.683915 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:14.683920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:14.684184 (+ 264us) server_negotiation.cc:407] Connection header received
0927 13:48:14.684346 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:14.684350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:14.684393 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:14.684471 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:14.685362 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:14.685366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:14.685369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:14.685381 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:14.685424 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:14.685438 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:14.685523 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:14.685685 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 13:48:14.685729 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 13:48:14.686697 23638 log.cc:824] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:14.689141 23638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e: No bootstrap required, opened a new log
I20260927 13:48:14.695680 23638 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } }
I20260927 13:48:14.696060 23638 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:14.696175 23638 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d111958b3b44e9b75ab7931cb7ac0e, State: Initialized, Role: FOLLOWER
I20260927 13:48:14.696786 23638 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } }
I20260927 13:48:14.696988 23638 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:14.697157 23638 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:14.697316 23638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:14.699424 23638 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } }
I20260927 13:48:14.699865 23638 leader_election.cc:302] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d111958b3b44e9b75ab7931cb7ac0e; no voters:
I20260927 13:48:14.700390 23638 leader_election.cc:288] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:14.700531 23648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:14.701243 23648 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [term 1 LEADER]: Becoming Leader. State: Replica: d1d111958b3b44e9b75ab7931cb7ac0e, State: Running, Role: LEADER
I20260927 13:48:14.701713 23648 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } }
I20260927 13:48:14.702261 23638 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:14.704423 23649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } } }
I20260927 13:48:14.704890 23649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:14.705417 23650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d111958b3b44e9b75ab7931cb7ac0e. Latest consensus state: current_term: 1 leader_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d111958b3b44e9b75ab7931cb7ac0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38591 } } }
I20260927 13:48:14.705711 23650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:14.705969 23656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:14.711314 23656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:14.719713 23656 catalog_manager.cc:1347] Generated new cluster ID: c54b272fadd34adfb5c8fba51845ecca
I20260927 13:48:14.719818 23656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:14.736071 23656 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:14.736497 23656 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:14.743110 23656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d111958b3b44e9b75ab7931cb7ac0e: Generated new TSK 0
I20260927 13:48:14.743793 23656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:14.750957 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23571
W20260927 13:48:14.752004 23633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.94:43057
W20260927 13:48:14.752589 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23571: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:14.752722 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23571
2026-09-27T13:48:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:14 dist-test-slave-7g3l krb5kdc[23669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:14 dist-test-slave-7g3l krb5kdc[23669](info): set up 2 sockets
Sep 27 13:48:14 dist-test-slave-7g3l krb5kdc[23669](info): commencing operation
krb5kdc: starting...
W20260927 13:48:15.819561 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:15 dist-test-slave-7g3l krb5kdc[23669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:16.035632 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36587
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:35629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:36587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:16.349557 23682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:16.349793 23682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:16.349876 23682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:16.356622 23682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:16.356711 23682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:16.356753 23682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:16.356812 23682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:16.356854 23682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:16.365679 23682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:35629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:36587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:36587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:16.366681 23682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:16.367826 23682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:16.378768 23687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:16.378803 23688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:16.378850 23690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:16.379710 23682 server_base.cc:1034] running on GCE node
I20260927 13:48:16.380384 23682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:16.381709 23682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:16.382928 23682 hybrid_clock.cc:648] HybridClock initialized: now 1790516896382890 us; error 47 us; skew 500 ppm
I20260927 13:48:16.383317 23682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:16 dist-test-slave-7g3l krb5kdc[23669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:16.386228 23682 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:16.387667 23682 webserver.cc:460] Webserver started at http://127.1.237.254:34659/ using document root <none> and password file <none>
I20260927 13:48:16.388302 23682 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:16.388442 23682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:16.388723 23682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:16.391553 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "da8e5a344ff941f793182d1093d05bec"
format_stamp: "Formatted at 2026-09-27 13:48:16 on dist-test-slave-7g3l"
server_key: "e7e1e4b952b5242edcb948306ca33524"
server_key_iv: "f2aca14b89bb0a433a6420af53f14442"
server_key_version: "encryptionkey@0"
I20260927 13:48:16.392287 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "da8e5a344ff941f793182d1093d05bec"
format_stamp: "Formatted at 2026-09-27 13:48:16 on dist-test-slave-7g3l"
server_key: "e7e1e4b952b5242edcb948306ca33524"
server_key_iv: "f2aca14b89bb0a433a6420af53f14442"
server_key_version: "encryptionkey@0"
I20260927 13:48:16.396881 23682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 13:48:16.400168 23697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:16.401347 23682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 13:48:16.401525 23682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "da8e5a344ff941f793182d1093d05bec"
format_stamp: "Formatted at 2026-09-27 13:48:16 on dist-test-slave-7g3l"
server_key: "e7e1e4b952b5242edcb948306ca33524"
server_key_iv: "f2aca14b89bb0a433a6420af53f14442"
server_key_version: "encryptionkey@0"
I20260927 13:48:16.401705 23682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:16.416653 23682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:16.420118 23682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:16.420375 23682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:16.437711 23682 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:36587
I20260927 13:48:16.437716 23748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:36587 every 8 connection(s)
I20260927 13:48:16.438946 23682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:16.442286 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:16.443843 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23682
I20260927 13:48:16.444029 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:16.444554 1975 external_mini_cluster.cc:1428] Setting key cdcbce93789f0e04f693621a46891f0e
I20260927 13:48:16.455937 23749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec: Bootstrap starting.
Sep 27 13:48:16 dist-test-slave-7g3l krb5kdc[23669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:48:16.459432 23749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:16.460595 23749 log.cc:824] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:16.463341 23749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec: No bootstrap required, opened a new log
I20260927 13:48:16.466468 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:16.446141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58635 (local address 127.1.237.254:36587)
0927 13:48:16.446931 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:16.446942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:16.446990 (+ 48us) server_negotiation.cc:407] Connection header received
0927 13:48:16.448561 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:16.448580 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:16.448969 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:16.449280 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:16.450955 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:16.452236 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:16.453969 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:16.454552 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:16.454669 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:16.459285 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:16.459293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:16.459310 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:16.459341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:16.462714 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:16.463475 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:16.463486 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:16.463491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:16.463646 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:16.464092 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:16.464103 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:16.464105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:16.464537 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:16.464886 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:16.465311 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 13:48:16.465582 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":401,"threads_started":1}
I20260927 13:48:16.469957 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:16.470981 23749 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } }
I20260927 13:48:16.471378 23749 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:16.471477 23749 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8e5a344ff941f793182d1093d05bec, State: Initialized, Role: FOLLOWER
I20260927 13:48:16.471976 23749 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } }
I20260927 13:48:16.472235 23749 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:16.472359 23749 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:16.472540 23749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:16.474952 23749 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } }
I20260927 13:48:16.475450 23749 leader_election.cc:302] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8e5a344ff941f793182d1093d05bec; no voters:
I20260927 13:48:16.476030 23749 leader_election.cc:288] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:16.476117 23759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:16.478308 23749 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:16.479440 23759 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [term 1 LEADER]: Becoming Leader. State: Replica: da8e5a344ff941f793182d1093d05bec, State: Running, Role: LEADER
I20260927 13:48:16.480149 23759 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } }
I20260927 13:48:16.483753 23761 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8e5a344ff941f793182d1093d05bec. Latest consensus state: current_term: 1 leader_uuid: "da8e5a344ff941f793182d1093d05bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } } }
I20260927 13:48:16.484133 23761 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:16.483852 23760 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8e5a344ff941f793182d1093d05bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8e5a344ff941f793182d1093d05bec" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 36587 } } }
I20260927 13:48:16.484241 23760 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:16.485062 23768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:16.491742 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:16.474122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.1.237.254:36587)
0927 13:48:16.474275 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:16.474281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:16.474661 (+ 380us) server_negotiation.cc:407] Connection header received
0927 13:48:16.474843 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:16.474847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:16.474910 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:16.475008 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:16.476536 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:16.477512 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:16.478785 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:16.479389 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:16.479542 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:16.482429 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:16.482436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:16.482444 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:16.482478 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:16.488786 (+ 6308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:16.489825 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:16.489828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:16.489830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:16.489942 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:16.490369 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:16.490374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:16.490378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:16.490644 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:16.490914 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:16.491294 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 13:48:16.491471 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 13:48:16.495195 23768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:16.503253 23768 catalog_manager.cc:1347] Generated new cluster ID: 2359bce9c918437396fb5f1d6f138712
I20260927 13:48:16.503356 23768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:16.533340 23768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:16.534633 23768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:16.551015 23768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8e5a344ff941f793182d1093d05bec: Generated new TSK 0
I20260927 13:48:16.551884 23768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:16.561389 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23682
I20260927 13:48:16.562652 23744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:16.651187 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23682
W20260927 13:48:16.664577 23667 connection.cc:535] client connection to 127.1.237.254:36587 recv error: Network error: failed to read from TLS socket (remote: 127.1.237.254:36587): Connection reset by peer (error 104)
2026-09-27T13:48:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:16 dist-test-slave-7g3l krb5kdc[23781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:16 dist-test-slave-7g3l krb5kdc[23781](info): set up 2 sockets
Sep 27 13:48:16 dist-test-slave-7g3l krb5kdc[23781](info): commencing operation
krb5kdc: starting...
W20260927 13:48:17.711552 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:17 dist-test-slave-7g3l krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:17.929852 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:40395
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:18.242581 23794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:18.242849 23794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:18.243053 23794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:18.250186 23794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:18.250283 23794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:18.250367 23794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:18.250475 23794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:18.250554 23794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:18.260044 23794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:40395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:18.261224 23794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:18.262423 23794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:18.273684 23799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:18.273591 23800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:18.273663 23802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:18.274497 23794 server_base.cc:1034] running on GCE node
I20260927 13:48:18.275133 23794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:18.276376 23794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:18.277590 23794 hybrid_clock.cc:648] HybridClock initialized: now 1790516898277548 us; error 56 us; skew 500 ppm
I20260927 13:48:18.277908 23794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:18 dist-test-slave-7g3l krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:18.280972 23794 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:18.282217 23794 webserver.cc:460] Webserver started at http://127.1.237.254:46675/ using document root <none> and password file <none>
I20260927 13:48:18.282811 23794 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:18.282938 23794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:18.283283 23794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:18.286088 23794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3"
format_stamp: "Formatted at 2026-09-27 13:48:18 on dist-test-slave-7g3l"
server_key: "94bb9af54fa881a283c0d92782f6f948"
server_key_iv: "f4ff4383de7b2786ab8a7a49f971982f"
server_key_version: "encryptionkey@0"
I20260927 13:48:18.286820 23794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3"
format_stamp: "Formatted at 2026-09-27 13:48:18 on dist-test-slave-7g3l"
server_key: "94bb9af54fa881a283c0d92782f6f948"
server_key_iv: "f4ff4383de7b2786ab8a7a49f971982f"
server_key_version: "encryptionkey@0"
I20260927 13:48:18.291564 23794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 13:48:18.294631 23809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:18.295866 23794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:48:18.296025 23794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3"
format_stamp: "Formatted at 2026-09-27 13:48:18 on dist-test-slave-7g3l"
server_key: "94bb9af54fa881a283c0d92782f6f948"
server_key_iv: "f4ff4383de7b2786ab8a7a49f971982f"
server_key_version: "encryptionkey@0"
I20260927 13:48:18.296192 23794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:18.318013 23794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:18.321775 23794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:18.322045 23794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:18.340090 23794 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:40395
I20260927 13:48:18.340122 23860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:40395 every 8 connection(s)
I20260927 13:48:18.341344 23794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:18.344858 23861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:18.351604 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23794
I20260927 13:48:18.351760 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:18.352195 1975 external_mini_cluster.cc:1428] Setting key be91b0df6582ab88a9eaf30da8dcd362
I20260927 13:48:18.354719 23861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3: Bootstrap starting.
I20260927 13:48:18.357911 23861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:18.358945 23861 log.cc:824] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:18.361706 23861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3: No bootstrap required, opened a new log
I20260927 13:48:18.364527 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:18.353823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40817 (local address 127.1.237.254:40395)
0927 13:48:18.354572 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:18.354583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:18.354620 (+ 37us) server_negotiation.cc:407] Connection header received
0927 13:48:18.356295 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:18.356316 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:18.356661 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:18.356949 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:18.357534 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:18.358824 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:18.360439 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:18.361050 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:18.361194 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:18.362385 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:18.362394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:18.362406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:18.362871 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:18.362943 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:18.362966 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:18.363073 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:18.363418 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 13:48:18.363613 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260927 13:48:18.368206 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:18.370129 23861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } }
I20260927 13:48:18.370471 23861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:18.370605 23861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a54a279cf3e4a838a1c4d9cb1ef2ca3, State: Initialized, Role: FOLLOWER
I20260927 13:48:18.371280 23861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } }
I20260927 13:48:18.371569 23861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:18.371743 23861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:18.371917 23861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:18.374878 23861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } }
I20260927 13:48:18.375397 23861 leader_election.cc:302] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a54a279cf3e4a838a1c4d9cb1ef2ca3; no voters:
I20260927 13:48:18.375965 23861 leader_election.cc:288] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:18.376091 23871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:18.377506 23871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 5a54a279cf3e4a838a1c4d9cb1ef2ca3, State: Running, Role: LEADER
I20260927 13:48:18.378203 23861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:18.378118 23871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } }
I20260927 13:48:18.380321 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:18.372461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57433 (local address 127.1.237.254:40395)
0927 13:48:18.372673 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:18.372683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:18.372962 (+ 279us) server_negotiation.cc:407] Connection header received
0927 13:48:18.373057 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:18.373064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:18.373140 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:18.373234 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:18.373641 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:18.374685 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:18.376410 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:18.377592 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:18.378361 (+ 769us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:18.378994 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:18.379004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:18.379008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:18.379026 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:18.379093 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:18.379112 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:18.379326 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:18.379806 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 13:48:18.379866 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:48:18.381402 23873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a54a279cf3e4a838a1c4d9cb1ef2ca3. Latest consensus state: current_term: 1 leader_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } } }
I20260927 13:48:18.381822 23873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:18.382454 23872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a54a279cf3e4a838a1c4d9cb1ef2ca3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 40395 } } }
I20260927 13:48:18.382798 23872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:18.383709 23878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:18.388646 23878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:18.397037 23878 catalog_manager.cc:1347] Generated new cluster ID: ae84d515415143cc91c85d3db9a09062
I20260927 13:48:18.397152 23878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:18.440122 23878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:18.441883 23878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:18.452747 23878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a54a279cf3e4a838a1c4d9cb1ef2ca3: Generated new TSK 0
I20260927 13:48:18.453532 23878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:18.460084 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23794
W20260927 13:48:18.461130 23856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40817
W20260927 13:48:18.461784 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:18.461930 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23794
W20260927 13:48:18.477998 23779 connection.cc:535] client connection to 127.1.237.254:40395 recv error: Network error: failed to read from TLS socket (remote: 127.1.237.254:40395): Connection reset by peer (error 104)
2026-09-27T13:48:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1813 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:18 dist-test-slave-7g3l krb5kdc[23892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:18 dist-test-slave-7g3l krb5kdc[23892](info): set up 2 sockets
Sep 27 13:48:18 dist-test-slave-7g3l krb5kdc[23892](info): commencing operation
krb5kdc: starting...
W20260927 13:48:19.531585 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:19 dist-test-slave-7g3l krb5kdc[23892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:19.754045 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43957
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:20.061496 23905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:20.061818 23905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:20.061976 23905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:20.068890 23905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:20.069033 23905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:20.069113 23905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:20.069196 23905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:20.069264 23905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:20.078974 23905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:43957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:20.080276 23905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:20.081619 23905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:20.092823 23913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:20.092823 23910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:20.092875 23911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:20.094000 23905 server_base.cc:1034] running on GCE node
I20260927 13:48:20.094748 23905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:20.096186 23905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:20.097384 23905 hybrid_clock.cc:648] HybridClock initialized: now 1790516900097342 us; error 29 us; skew 500 ppm
I20260927 13:48:20.097733 23905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:20 dist-test-slave-7g3l krb5kdc[23892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516900, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:20.105845 23905 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:20.107370 23905 webserver.cc:460] Webserver started at http://127.1.237.254:38275/ using document root <none> and password file <none>
I20260927 13:48:20.108017 23905 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:20.108150 23905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:20.108460 23905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:20.111395 23905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "b45b07e3f2bc4c29839e32267385eac1"
format_stamp: "Formatted at 2026-09-27 13:48:20 on dist-test-slave-7g3l"
server_key: "965a75c4d5451f0468950c8e19a158e1"
server_key_iv: "ea6862519c31a7aa8b84103e36f7c64a"
server_key_version: "encryptionkey@0"
I20260927 13:48:20.112161 23905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "b45b07e3f2bc4c29839e32267385eac1"
format_stamp: "Formatted at 2026-09-27 13:48:20 on dist-test-slave-7g3l"
server_key: "965a75c4d5451f0468950c8e19a158e1"
server_key_iv: "ea6862519c31a7aa8b84103e36f7c64a"
server_key_version: "encryptionkey@0"
I20260927 13:48:20.116838 23905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:48:20.119971 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:20.121121 23905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:48:20.121289 23905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "b45b07e3f2bc4c29839e32267385eac1"
format_stamp: "Formatted at 2026-09-27 13:48:20 on dist-test-slave-7g3l"
server_key: "965a75c4d5451f0468950c8e19a158e1"
server_key_iv: "ea6862519c31a7aa8b84103e36f7c64a"
server_key_version: "encryptionkey@0"
I20260927 13:48:20.121469 23905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:20.126668 23905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:20.130373 23905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:20.130661 23905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:20.148556 23905 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:43957
I20260927 13:48:20.148594 23971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:43957 every 8 connection(s)
I20260927 13:48:20.149844 23905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:20.152640 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 23905
I20260927 13:48:20.152796 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:20.153231 1975 external_mini_cluster.cc:1428] Setting key bc705feeff6f352e42bf26a4338b72cb
I20260927 13:48:20.153606 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:20.161824 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:20.154938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42905 (local address 127.1.237.254:43957)
0927 13:48:20.155788 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:20.155802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:20.155851 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:48:20.158198 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:20.158224 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:20.158648 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:20.159059 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:20.160020 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:20.160029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:20.160052 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:20.160073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:20.160134 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:20.160170 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:20.160388 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:20.160515 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 13:48:20.160688 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":354,"threads_started":1}
I20260927 13:48:20.165241 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:20.167382 23972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1: Bootstrap starting.
I20260927 13:48:20.170890 23972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:20.171816 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:20.169243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42279 (local address 127.1.237.254:43957)
0927 13:48:20.169448 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:20.169456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:20.169900 (+ 444us) server_negotiation.cc:407] Connection header received
0927 13:48:20.170057 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:20.170060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:20.170100 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:20.170173 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:20.171121 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:20.171125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:20.171128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:20.171140 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:20.171243 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:20.171259 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:20.171351 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:20.171501 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 13:48:20.171541 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:48:20.172263 23972 log.cc:824] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:20.174750 23972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1: No bootstrap required, opened a new log
I20260927 13:48:20.181552 23972 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } }
I20260927 13:48:20.181919 23972 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:20.182020 23972 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45b07e3f2bc4c29839e32267385eac1, State: Initialized, Role: FOLLOWER
I20260927 13:48:20.182513 23972 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } }
I20260927 13:48:20.182736 23972 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:20.182881 23972 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:20.183044 23972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:20.185056 23972 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } }
I20260927 13:48:20.185563 23972 leader_election.cc:302] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45b07e3f2bc4c29839e32267385eac1; no voters:
I20260927 13:48:20.186131 23972 leader_election.cc:288] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:20.186216 23982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:20.187045 23982 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [term 1 LEADER]: Becoming Leader. State: Replica: b45b07e3f2bc4c29839e32267385eac1, State: Running, Role: LEADER
I20260927 13:48:20.187637 23982 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } }
I20260927 13:48:20.188308 23972 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:20.190112 23983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45b07e3f2bc4c29839e32267385eac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } } }
I20260927 13:48:20.190448 23983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:20.192452 23988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:20.192394 23984 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45b07e3f2bc4c29839e32267385eac1. Latest consensus state: current_term: 1 leader_uuid: "b45b07e3f2bc4c29839e32267385eac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b07e3f2bc4c29839e32267385eac1" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 43957 } } }
I20260927 13:48:20.192703 23984 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:20.197708 23988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:20.205281 23988 catalog_manager.cc:1347] Generated new cluster ID: 8b358e8d48e846928ebc5e6742970648
I20260927 13:48:20.205385 23988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:20.221686 23988 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:20.222091 23988 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:20.228675 23988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45b07e3f2bc4c29839e32267385eac1: Generated new TSK 0
I20260927 13:48:20.229368 23988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:20.233807 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23905
W20260927 13:48:20.234855 23967 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42905
W20260927 13:48:20.235512 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 23905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:20.235656 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 23905
2026-09-27T13:48:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1773 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:20 dist-test-slave-7g3l krb5kdc[24003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:20 dist-test-slave-7g3l krb5kdc[24003](info): set up 2 sockets
Sep 27 13:48:20 dist-test-slave-7g3l krb5kdc[24003](info): commencing operation
krb5kdc: starting...
W20260927 13:48:21.299644 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:21 dist-test-slave-7g3l krb5kdc[24003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:21.521720 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:44987
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:21.828373 24016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:21.828691 24016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:21.828871 24016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:21.835450 24016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:21.835566 24016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:21.835646 24016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:21.835759 24016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:21.835832 24016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:21.844920 24016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:44987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:21.846057 24016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:21.847335 24016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:21.858737 24021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:21.858693 24022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:21.858771 24024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:21.859499 24016 server_base.cc:1034] running on GCE node
I20260927 13:48:21.860196 24016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:21.861409 24016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:21.862654 24016 hybrid_clock.cc:648] HybridClock initialized: now 1790516901862608 us; error 43 us; skew 500 ppm
I20260927 13:48:21.862977 24016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:21 dist-test-slave-7g3l krb5kdc[24003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516901, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:21.865794 24016 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:21.866994 24016 webserver.cc:460] Webserver started at http://127.1.237.254:41965/ using document root <none> and password file <none>
I20260927 13:48:21.867609 24016 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:21.867720 24016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:21.867971 24016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:21.870750 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "427cffe2741247aa94d316f124b1bbd4"
format_stamp: "Formatted at 2026-09-27 13:48:21 on dist-test-slave-7g3l"
server_key: "cf4a7b2389c09282aa4a943001b7a7a9"
server_key_iv: "294696019780d1ac8f6dcf9eb313b275"
server_key_version: "encryptionkey@0"
I20260927 13:48:21.871523 24016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "427cffe2741247aa94d316f124b1bbd4"
format_stamp: "Formatted at 2026-09-27 13:48:21 on dist-test-slave-7g3l"
server_key: "cf4a7b2389c09282aa4a943001b7a7a9"
server_key_iv: "294696019780d1ac8f6dcf9eb313b275"
server_key_version: "encryptionkey@0"
I20260927 13:48:21.876175 24016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 13:48:21.879045 24031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:21.880128 24016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 13:48:21.880286 24016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "427cffe2741247aa94d316f124b1bbd4"
format_stamp: "Formatted at 2026-09-27 13:48:21 on dist-test-slave-7g3l"
server_key: "cf4a7b2389c09282aa4a943001b7a7a9"
server_key_iv: "294696019780d1ac8f6dcf9eb313b275"
server_key_version: "encryptionkey@0"
I20260927 13:48:21.880448 24016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:21.898428 24016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:21.902356 24016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:21.902617 24016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:21.920022 24016 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:44987
I20260927 13:48:21.920050 24082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:44987 every 8 connection(s)
I20260927 13:48:21.921239 24016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:21.922646 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24016
I20260927 13:48:21.922798 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:21.923411 1975 external_mini_cluster.cc:1428] Setting key e5605109a3eab8a88060be1a2b9d8d83
I20260927 13:48:21.925486 24083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 13:48:21 dist-test-slave-7g3l krb5kdc[24003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:48:21.937965 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: Bootstrap starting.
I20260927 13:48:21.941850 24083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:21.942976 24083 log.cc:824] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:21.946063 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: No bootstrap required, opened a new log
I20260927 13:48:21.946074 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:21.925075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55397 (local address 127.1.237.254:44987)
0927 13:48:21.925842 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:21.925852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:21.925888 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:48:21.927394 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:21.927412 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:21.927794 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:21.928094 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:21.929760 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:21.931055 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:21.932735 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:21.933349 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:21.933472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:21.938061 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:21.938073 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:21.938088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:21.938133 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:21.941725 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:21.942456 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:21.942464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:21.942471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:21.942671 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:21.943074 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:21.943080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:21.943083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:21.943730 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:21.943989 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:21.944463 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 13:48:21.944839 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":366,"threads_started":1}
I20260927 13:48:21.950014 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:21.954614 24083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } }
I20260927 13:48:21.954955 24083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:21.955070 24083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427cffe2741247aa94d316f124b1bbd4, State: Initialized, Role: FOLLOWER
I20260927 13:48:21.955647 24083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } }
I20260927 13:48:21.955883 24083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:21.956014 24083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:21.956190 24083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:21.958312 24083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } }
I20260927 13:48:21.958778 24083 leader_election.cc:302] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427cffe2741247aa94d316f124b1bbd4; no voters:
I20260927 13:48:21.959438 24083 leader_election.cc:288] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:21.959537 24093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:21.961720 24083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:21.966104 24093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [term 1 LEADER]: Becoming Leader. State: Replica: 427cffe2741247aa94d316f124b1bbd4, State: Running, Role: LEADER
I20260927 13:48:21.966815 24093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } }
I20260927 13:48:21.967739 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:21.954020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50877 (local address 127.1.237.254:44987)
0927 13:48:21.954173 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:21.954180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:21.954573 (+ 393us) server_negotiation.cc:407] Connection header received
0927 13:48:21.954739 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:21.954744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:21.954802 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:21.954902 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:21.956444 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:21.957147 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:21.958333 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:21.958786 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:21.958931 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:21.961360 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:21.961365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:21.961368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:21.961399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:21.964894 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:21.965470 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:21.965477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:21.965481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:21.965705 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:21.966113 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:21.966119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:21.966123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:21.966541 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:21.966867 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:21.967177 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 13:48:21.967365 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 13:48:21.972867 24094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427cffe2741247aa94d316f124b1bbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } } }
I20260927 13:48:21.973416 24094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:21.973857 24095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427cffe2741247aa94d316f124b1bbd4. Latest consensus state: current_term: 1 leader_uuid: "427cffe2741247aa94d316f124b1bbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427cffe2741247aa94d316f124b1bbd4" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 44987 } } }
I20260927 13:48:21.974632 24095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4 [sys.catalog]: This master's current role is: LEADER
W20260927 13:48:21.977173 24107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:48:21.977275 24107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:48:21.977417 24108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:21.979199 24108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:21.986742 24108 catalog_manager.cc:1347] Generated new cluster ID: 7de004afed5b48c9aafaf21e0ff45a53
I20260927 13:48:21.986852 24108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:22.013491 24108 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:22.015254 24108 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:22.026695 24108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427cffe2741247aa94d316f124b1bbd4: Generated new TSK 0
I20260927 13:48:22.027537 24108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:22.041746 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24016
I20260927 13:48:22.042716 24078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:22.132901 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24016
2026-09-27T13:48:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:22 dist-test-slave-7g3l krb5kdc[24115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:22 dist-test-slave-7g3l krb5kdc[24115](info): set up 2 sockets
Sep 27 13:48:22 dist-test-slave-7g3l krb5kdc[24115](info): commencing operation
krb5kdc: starting...
W20260927 13:48:23.219535 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:23 dist-test-slave-7g3l krb5kdc[24115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:23.462522 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34911
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:23.784123 24128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:23.784430 24128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:23.784595 24128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:23.791698 24128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:23.791827 24128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:23.791937 24128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:23.792037 24128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:23.792120 24128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:23.801350 24128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:23.802573 24128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:23.803844 24128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:23.815191 24133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:23.815125 24134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:23.815402 24136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:23.816138 24128 server_base.cc:1034] running on GCE node
I20260927 13:48:23.816918 24128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:23.818259 24128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:23.819483 24128 hybrid_clock.cc:648] HybridClock initialized: now 1790516903819452 us; error 46 us; skew 500 ppm
I20260927 13:48:23.819825 24128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:23 dist-test-slave-7g3l krb5kdc[24115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516903, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:23.823331 24128 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:23.824630 24128 webserver.cc:460] Webserver started at http://127.1.237.254:40793/ using document root <none> and password file <none>
I20260927 13:48:23.825220 24128 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:23.825316 24128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:23.825556 24128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:23.828462 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "d12a095e6e744552bbc8ee44a81750e3"
format_stamp: "Formatted at 2026-09-27 13:48:23 on dist-test-slave-7g3l"
server_key: "c60af35e1f0240943bacee76af83470d"
server_key_iv: "d5c7b37a4a0f621550c2000d28ace3ce"
server_key_version: "encryptionkey@0"
I20260927 13:48:23.829167 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "d12a095e6e744552bbc8ee44a81750e3"
format_stamp: "Formatted at 2026-09-27 13:48:23 on dist-test-slave-7g3l"
server_key: "c60af35e1f0240943bacee76af83470d"
server_key_iv: "d5c7b37a4a0f621550c2000d28ace3ce"
server_key_version: "encryptionkey@0"
I20260927 13:48:23.833963 24128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 13:48:23.837172 24143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:23.838460 24128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:48:23.838650 24128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "d12a095e6e744552bbc8ee44a81750e3"
format_stamp: "Formatted at 2026-09-27 13:48:23 on dist-test-slave-7g3l"
server_key: "c60af35e1f0240943bacee76af83470d"
server_key_iv: "d5c7b37a4a0f621550c2000d28ace3ce"
server_key_version: "encryptionkey@0"
I20260927 13:48:23.838848 24128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:23.856385 24128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:23.860028 24128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:23.860282 24128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:23.877928 24128 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:34911
I20260927 13:48:23.877966 24194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:34911 every 8 connection(s)
I20260927 13:48:23.879220 24128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:23.882663 24195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:23.883181 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24128
I20260927 13:48:23.883352 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:23.883776 1975 external_mini_cluster.cc:1428] Setting key ec20d97435286abe1186c45c85a96d27
I20260927 13:48:23.896489 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3: Bootstrap starting.
I20260927 13:48:23.898300 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:23.885519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49713 (local address 127.1.237.254:34911)
0927 13:48:23.886283 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:23.886302 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:23.886353 (+ 51us) server_negotiation.cc:407] Connection header received
0927 13:48:23.889055 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:23.889081 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:23.889549 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:23.889973 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:23.890429 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:23.892266 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:23.893535 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:23.894551 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:23.894705 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:23.895787 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:23.895797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:23.895812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:23.896491 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:23.896584 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:23.896613 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:23.896726 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:23.896868 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 13:48:23.897100 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":317,"threads_started":1}
I20260927 13:48:23.899952 24195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:23.901165 24195 log.cc:824] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:23.901784 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:23.903991 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3: No bootstrap required, opened a new log
I20260927 13:48:23.912457 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:23.906613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49705 (local address 127.1.237.254:34911)
0927 13:48:23.906831 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:23.906838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:23.907129 (+ 291us) server_negotiation.cc:407] Connection header received
0927 13:48:23.907341 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:23.907345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:23.907403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:23.907517 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:23.907972 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:23.908803 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:23.910080 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:23.910556 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:23.910708 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:23.911754 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:23.911762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:23.911770 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:23.911786 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:23.911836 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:23.911855 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:23.911958 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:23.912089 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 13:48:23.912171 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 13:48:23.912788 24195 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } }
I20260927 13:48:23.913153 24195 raft_consensus.cc:374] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:23.913270 24195 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12a095e6e744552bbc8ee44a81750e3, State: Initialized, Role: FOLLOWER
I20260927 13:48:23.913847 24195 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } }
I20260927 13:48:23.914135 24195 raft_consensus.cc:388] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:23.914283 24195 raft_consensus.cc:482] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:23.914458 24195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:23.916610 24195 raft_consensus.cc:504] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } }
I20260927 13:48:23.917049 24195 leader_election.cc:302] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d12a095e6e744552bbc8ee44a81750e3; no voters:
I20260927 13:48:23.917724 24195 leader_election.cc:288] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:23.917876 24205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:23.918514 24205 raft_consensus.cc:686] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [term 1 LEADER]: Becoming Leader. State: Replica: d12a095e6e744552bbc8ee44a81750e3, State: Running, Role: LEADER
I20260927 13:48:23.919046 24205 consensus_queue.cc:237] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } }
I20260927 13:48:23.920857 24205 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d12a095e6e744552bbc8ee44a81750e3. Latest consensus state: current_term: 1 leader_uuid: "d12a095e6e744552bbc8ee44a81750e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } } }
I20260927 13:48:23.921118 24205 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:23.921764 24207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d12a095e6e744552bbc8ee44a81750e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12a095e6e744552bbc8ee44a81750e3" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34911 } } }
I20260927 13:48:23.922132 24207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:23.924471 24195 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:23.925954 24209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:23.931080 24209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:23.939916 24209 catalog_manager.cc:1347] Generated new cluster ID: 7f9907ffd08a4eefa8f2853d12632a86
I20260927 13:48:23.940009 24209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:23.959393 24209 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:23.960520 24209 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:23.969004 24209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d12a095e6e744552bbc8ee44a81750e3: Generated new TSK 0
I20260927 13:48:23.969681 24209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:23.977209 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24128
W20260927 13:48:23.978215 24190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49713
W20260927 13:48:23.978786 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:23.978942 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24128
2026-09-27T13:48:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:24 dist-test-slave-7g3l krb5kdc[24226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:24 dist-test-slave-7g3l krb5kdc[24226](info): set up 2 sockets
Sep 27 13:48:24 dist-test-slave-7g3l krb5kdc[24226](info): commencing operation
krb5kdc: starting...
W20260927 13:48:25.059700 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:25 dist-test-slave-7g3l krb5kdc[24226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:25.276468 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34841
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:34893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:25.589031 24239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:25.589289 24239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:25.589449 24239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:25.596410 24239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:25.596525 24239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:25.596623 24239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:25.596721 24239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:25.596805 24239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:25.605822 24239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:34893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:25.607002 24239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:25.608215 24239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:25.619489 24244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:25.619544 24245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:25.619449 24247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:25.620338 24239 server_base.cc:1034] running on GCE node
I20260927 13:48:25.621017 24239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:25.622233 24239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:25.623422 24239 hybrid_clock.cc:648] HybridClock initialized: now 1790516905623384 us; error 29 us; skew 500 ppm
I20260927 13:48:25.623737 24239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:25 dist-test-slave-7g3l krb5kdc[24226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:25.632383 24239 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:25.633630 24239 webserver.cc:460] Webserver started at http://127.1.237.254:43963/ using document root <none> and password file <none>
I20260927 13:48:25.634209 24239 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:25.634299 24239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:25.634529 24239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:25.637439 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "829b74bcad9646a7b8e49c3d17c4de82"
format_stamp: "Formatted at 2026-09-27 13:48:25 on dist-test-slave-7g3l"
server_key: "296d1491976972a0cf16553895d1922d"
server_key_iv: "5ba80e58009bdfa14b830bb322370ffd"
server_key_version: "encryptionkey@0"
I20260927 13:48:25.638127 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "829b74bcad9646a7b8e49c3d17c4de82"
format_stamp: "Formatted at 2026-09-27 13:48:25 on dist-test-slave-7g3l"
server_key: "296d1491976972a0cf16553895d1922d"
server_key_iv: "5ba80e58009bdfa14b830bb322370ffd"
server_key_version: "encryptionkey@0"
I20260927 13:48:25.642702 24239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 13:48:25.645982 24254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:25.647200 24239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 13:48:25.647362 24239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "829b74bcad9646a7b8e49c3d17c4de82"
format_stamp: "Formatted at 2026-09-27 13:48:25 on dist-test-slave-7g3l"
server_key: "296d1491976972a0cf16553895d1922d"
server_key_iv: "5ba80e58009bdfa14b830bb322370ffd"
server_key_version: "encryptionkey@0"
I20260927 13:48:25.647564 24239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:25.652694 24239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:25.656253 24239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:25.656528 24239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:25.673741 24239 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:34841
I20260927 13:48:25.673774 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:34841 every 8 connection(s)
I20260927 13:48:25.675051 24239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:25.678463 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:25.684814 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24239
I20260927 13:48:25.685003 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:25.685539 1975 external_mini_cluster.cc:1428] Setting key 03473ebbbd43588ae53c7f12bffbb807
I20260927 13:48:25.689080 24306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82: Bootstrap starting.
I20260927 13:48:25.692301 24306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:25.692838 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:25.687271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51013 (local address 127.1.237.254:34841)
0927 13:48:25.687999 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:25.688009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:25.688047 (+ 38us) server_negotiation.cc:407] Connection header received
0927 13:48:25.689712 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:25.689732 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:25.690040 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:25.690405 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:25.691386 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:25.691392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:25.691403 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:25.691420 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:25.691467 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:25.691484 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:25.691663 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:25.691795 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 13:48:25.691967 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":308,"threads_started":1}
I20260927 13:48:25.693367 24306 log.cc:824] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:25.696249 24306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82: No bootstrap required, opened a new log
I20260927 13:48:25.696280 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:25.702719 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:25.700320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40997 (local address 127.1.237.254:34841)
0927 13:48:25.700466 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:25.700470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:25.700897 (+ 427us) server_negotiation.cc:407] Connection header received
0927 13:48:25.701073 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:25.701078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:25.701134 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:25.701234 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:25.702075 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:25.702080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:25.702085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 13:48:25.702101 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:25.702144 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 13:48:25.702158 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:25.702286 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:25.702404 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 13:48:25.702466 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 13:48:25.704457 24306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } }
I20260927 13:48:25.704828 24306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 13:48:25.704938 24306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829b74bcad9646a7b8e49c3d17c4de82, State: Initialized, Role: FOLLOWER
I20260927 13:48:25.705504 24306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } }
I20260927 13:48:25.705747 24306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 13:48:25.705901 24306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 13:48:25.706076 24306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:25.708473 24306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } }
I20260927 13:48:25.708942 24306 leader_election.cc:302] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 829b74bcad9646a7b8e49c3d17c4de82; no voters:
I20260927 13:48:25.709532 24306 leader_election.cc:288] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 13:48:25.709647 24316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:25.710350 24316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [term 1 LEADER]: Becoming Leader. State: Replica: 829b74bcad9646a7b8e49c3d17c4de82, State: Running, Role: LEADER
I20260927 13:48:25.711282 24316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } }
I20260927 13:48:25.711803 24306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:25.713913 24317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "829b74bcad9646a7b8e49c3d17c4de82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } } }
I20260927 13:48:25.714102 24318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 829b74bcad9646a7b8e49c3d17c4de82. Latest consensus state: current_term: 1 leader_uuid: "829b74bcad9646a7b8e49c3d17c4de82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829b74bcad9646a7b8e49c3d17c4de82" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 34841 } } }
I20260927 13:48:25.714386 24318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:25.714386 24317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:25.715210 24323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:25.721984 24323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:25.732883 24323 catalog_manager.cc:1347] Generated new cluster ID: ff3fc30a54bf4d44bb1b07f6517a95a5
I20260927 13:48:25.733119 24323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:25.749176 24323 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:25.749593 24323 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:25.758953 24323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 829b74bcad9646a7b8e49c3d17c4de82: Generated new TSK 0
I20260927 13:48:25.759742 24323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:25.769739 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24239
W20260927 13:48:25.770768 24301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51013
W20260927 13:48:25.771426 1975 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 13:48:25.771566 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24239
2026-09-27T13:48:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1793 ms)
[----------] 18 tests from AuthTokenIssuingTest (33070 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:25 dist-test-slave-7g3l krb5kdc[24337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:25 dist-test-slave-7g3l krb5kdc[24337](info): set up 2 sockets
Sep 27 13:48:25 dist-test-slave-7g3l krb5kdc[24337](info): commencing operation
krb5kdc: starting...
W20260927 13:48:26.847783 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:26 dist-test-slave-7g3l krb5kdc[24337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:27.075661 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45931
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:27.385133 24350 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:27.385507 24350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:27.385638 24350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:27.385779 24350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:27.393013 24350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:27.393113 24350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:27.393213 24350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:27.393293 24350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:27.393384 24350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:27.402541 24350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:45931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:27.403667 24350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:27.404845 24350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:27.415915 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:27.416074 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:27.416137 24358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:27.416826 24350 server_base.cc:1034] running on GCE node
I20260927 13:48:27.417503 24350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:27.418744 24350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:27.419975 24350 hybrid_clock.cc:648] HybridClock initialized: now 1790516907419913 us; error 39 us; skew 500 ppm
I20260927 13:48:27.420285 24350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:27 dist-test-slave-7g3l krb5kdc[24337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:27.423251 24350 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:27.424541 24350 webserver.cc:460] Webserver started at http://127.1.237.254:37505/ using document root <none> and password file <none>
I20260927 13:48:27.425117 24350 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:27.425196 24350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:27.425468 24350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:27.428246 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "c95fa3369f6c4e1388ee22de6bcd9e84"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "c5d0bed74c47026d4e670c5f92ef01f9"
server_key_iv: "d79d5627fbb6e06d74ac84d67fb5c213"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.428927 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "c95fa3369f6c4e1388ee22de6bcd9e84"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "c5d0bed74c47026d4e670c5f92ef01f9"
server_key_iv: "d79d5627fbb6e06d74ac84d67fb5c213"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.433517 24350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 13:48:27.437139 24365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:27.438895 24350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 13:48:27.439059 24350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "c95fa3369f6c4e1388ee22de6bcd9e84"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "c5d0bed74c47026d4e670c5f92ef01f9"
server_key_iv: "d79d5627fbb6e06d74ac84d67fb5c213"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.439317 24350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:27.452307 24350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:27.455752 24350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:27.456000 24350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:27.473353 24350 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:45931
I20260927 13:48:27.473371 24416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:45931 every 8 connection(s)
I20260927 13:48:27.474576 24350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:27.476162 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24350
I20260927 13:48:27.476382 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:27.477124 1975 external_mini_cluster.cc:1428] Setting key effa94fd666d2847644d2675b8c52bd3
I20260927 13:48:27.478852 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:27.488101 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.237.253@KRBTEST.COM; defaulting to no policy
Sep 27 13:48:27 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.253@KRBTEST.COM" created.
I20260927 13:48:27.510572 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.491007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45931 (local address 127.0.0.1:34146)
0927 13:48:27.492048 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.492085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:27.493685 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:27.494429 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:27.494435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:27.495062 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:27.495447 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.495465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.496917 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.496923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.498019 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.498030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.498893 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.498899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.499066 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.499723 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:27.499744 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:27.501862 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:27.505922 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:27.505935 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:27.505947 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:27.506450 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:27.506950 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:27.506957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:27.506964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:27.507144 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:27.508166 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:27.508176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:27.508425 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 13:48:27.508670 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 13:48:27.509195 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"spinlock_wait_cycles":24832,"thread_start_us":660,"threads_started":1}
I20260927 13:48:27.511721 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.491000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.1.237.254:45931)
0927 13:48:27.491780 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:27.491798 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:27.492192 (+ 394us) server_negotiation.cc:407] Connection header received
0927 13:48:27.494015 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:27.494020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:27.494215 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:27.494404 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:27.495644 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.496764 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.498159 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.498663 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.498843 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.502068 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:27.502075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:27.502084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:27.502110 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:27.505782 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:27.506605 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:27.506616 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:27.506622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:27.506805 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:27.507283 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:27.507291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:27.507297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:27.507925 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:27.508108 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:27.510972 (+ 2864us) server_negotiation.cc:299] Negotiation successful
0927 13:48:27.511242 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
I20260927 13:48:27.514539 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:27.515475 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.515119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38052)
0927 13:48:27.515291 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.515383 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 13:48:27.515726 24367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
I20260927 13:48:27.517148 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.516834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38062)
0927 13:48:27.516975 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.517036 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 13:48:27.517313 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:44299: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
I20260927 13:48:27.532048 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:44299
--webserver_interface=127.1.237.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:48:27.566582 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } attempt: 1
I20260927 13:48:27.567638 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.567227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38078)
0927 13:48:27.567405 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.567522 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:48:27.568527 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.568211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38086)
0927 13:48:27.568344 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.568413 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 13:48:27.568704 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:44299: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
I20260927 13:48:27.636971 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } attempt: 2
I20260927 13:48:27.638175 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.637652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38096)
0927 13:48:27.637945 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.638031 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 13:48:27.639115 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.638778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38104)
0927 13:48:27.638922 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.639001 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:48:27.639400 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:44299: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
I20260927 13:48:27.730643 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } attempt: 3
I20260927 13:48:27.731809 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.731399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38106)
0927 13:48:27.731587 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.731682 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:48:27.732837 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.732473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38112)
0927 13:48:27.732641 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.732709 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 13:48:27.733031 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:44299: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
W20260927 13:48:27.839612 24424 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:27.839967 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:27.840086 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:27.840265 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:27.847440 24424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:27.847520 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:27.847631 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:27.847715 24424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:27.847796 24424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:27.857431 24424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.237.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:27.858630 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:27.859927 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:27.871271 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:27.871369 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:27.871315 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:27.872087 24424 server_base.cc:1034] running on GCE node
I20260927 13:48:27.872860 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:27.874168 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:27.875375 24424 hybrid_clock.cc:648] HybridClock initialized: now 1790516907875315 us; error 31 us; skew 500 ppm
I20260927 13:48:27.875687 24424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 13:48:27.876230 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } attempt: 4
I20260927 13:48:27.877394 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.876885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38114)
0927 13:48:27.877184 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.877259 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 13:48:27.878294 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.877959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38122)
0927 13:48:27.878099 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.878185 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Sep 27 13:48:27 dist-test-slave-7g3l krb5kdc[24337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 13:48:27.878573 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:44299: Network error: Client connection negotiation failed: client connection to 127.1.237.253:44299: connect: Connection refused (error 111)
I20260927 13:48:27.879060 24424 init.cc:377] Logged in from keytab as kudu/127.1.237.253@KRBTEST.COM (short username kudu)
I20260927 13:48:27.880453 24424 webserver.cc:460] Webserver started at http://127.1.237.253:33909/ using document root <none> and password file <none>
I20260927 13:48:27.881090 24424 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:27.881196 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:27.881505 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:27.884284 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data/instance:
uuid: "7ef8e3410acb4b85b16d57fda5fc7e02"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "6a653aac2cc64d7db0458a579d77c942"
server_key_iv: "b960ab9d2e91370ef82b01afa4f282eb"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.885033 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal/instance:
uuid: "7ef8e3410acb4b85b16d57fda5fc7e02"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "6a653aac2cc64d7db0458a579d77c942"
server_key_iv: "b960ab9d2e91370ef82b01afa4f282eb"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.889842 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 13:48:27.892941 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:27.894164 24424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 13:48:27.894354 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "7ef8e3410acb4b85b16d57fda5fc7e02"
format_stamp: "Formatted at 2026-09-27 13:48:27 on dist-test-slave-7g3l"
server_key: "6a653aac2cc64d7db0458a579d77c942"
server_key_iv: "b960ab9d2e91370ef82b01afa4f282eb"
server_key_version: "encryptionkey@0"
I20260927 13:48:27.894569 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:27.931627 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:27.935357 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:27.935640 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:27.953747 24424 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.253:44299
I20260927 13:48:27.953755 24490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.253:44299 every 8 connection(s)
I20260927 13:48:27.955099 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
I20260927 13:48:27.958711 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:27.962330 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24424
I20260927 13:48:27.962522 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-1/wal/instance
I20260927 13:48:27.963136 1975 external_mini_cluster.cc:1428] Setting key 404f108606ec67579a6fa07db75de368
I20260927 13:48:27.964982 24491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:27 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:48:27.985001 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.967538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.1.237.254:45931)
0927 13:48:27.967709 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:27.967714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:27.968926 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 13:48:27.970737 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:27.970742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:27.970841 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:27.970959 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:27.972414 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.973080 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.974600 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.975001 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.975134 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.978982 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:27.978987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:27.978990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:27.979022 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:27.981663 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:27.982508 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:27.982516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:27.982522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:27.982676 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:27.983090 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:27.983098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:27.983102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:27.983557 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:27.983720 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:27.984359 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 13:48:27.984611 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 13:48:27.985601 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.968062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45931 (local address 127.0.0.1:34152)
0927 13:48:27.968757 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.968792 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:27.970475 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:27.971248 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:27.971256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:27.971741 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:27.972252 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.972268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.973258 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.973262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.974439 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.974451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.975241 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.975249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.975475 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.976135 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:27.976161 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:27.978766 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:27.981813 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:27.981821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:27.981837 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:27.982387 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:27.982791 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:27.982794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:27.982796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:27.982973 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:27.983680 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:27.983685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:27.983992 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 13:48:27.984225 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 13:48:27.984573 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":301,"threads_started":1}
WARNING: no policy specified for kudu/127.1.237.252@KRBTEST.COM; defaulting to no policy
I20260927 13:48:27.987403 24491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.252@KRBTEST.COM" created.
Sep 27 13:48:27 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:28.002830 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.988298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.1.237.253:44299)
0927 13:48:27.988981 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:27.988989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:27.989021 (+ 32us) server_negotiation.cc:407] Connection header received
0927 13:48:27.989078 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:27.989087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:27.989313 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:27.989473 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:27.990127 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.991880 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.992969 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.993867 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.994006 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.996903 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:27.996918 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:27.996942 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:27.996988 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.000037 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.000803 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.000808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.000810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.000910 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.001361 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.001364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.001366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.001669 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.001816 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.002239 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.002545 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":505,"thread_start_us":334,"threads_started":1}
I20260927 13:48:28.002936 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:27.987992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38126)
0927 13:48:27.988153 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:27.988181 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:27.988287 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:27.989469 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:27.989472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:27.989803 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:27.989953 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.989961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.992008 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.992011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.992819 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:27.992829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:27.994001 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:27.994005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:27.994136 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:27.994693 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:27.994709 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:27.996683 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.000160 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.000169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.000176 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.000689 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.001032 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.001040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.001046 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.001260 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.001792 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.001800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.001967 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.002121 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.002432 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 13:48:28.005228 24491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:28.006346 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.005889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37330)
0927 13:48:28.006041 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.006206 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 13:48:28.006569 24442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.008060 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.007754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37340)
0927 13:48:28.007884 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.007967 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 13:48:28.008301 24491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
I20260927 13:48:28.020419 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:34691
--webserver_interface=127.1.237.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:48:28.057595 24491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 1
I20260927 13:48:28.058724 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.058267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37354)
0927 13:48:28.058483 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.058584 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 13:48:28.059686 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.059358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37360)
0927 13:48:28.059526 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.059589 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 13:48:28.059890 24491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.128086 24491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 2
I20260927 13:48:28.129216 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.128706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37364)
0927 13:48:28.128921 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.129043 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 13:48:28.130151 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.129822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37366)
0927 13:48:28.129979 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.130047 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:48:28.130336 24491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.177894 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } attempt: 5
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:28.192444 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.178727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.1.237.253:44299)
0927 13:48:28.178909 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.178914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.178939 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:48:28.179143 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.179163 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.179223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.179319 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.180256 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.180963 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.182627 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.183040 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.183128 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.187273 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.187278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.187281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.187311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.190008 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.190735 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.190739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.190741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.190859 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.191249 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.191252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.191254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.191523 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.191630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.192070 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.192223 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:48:28.192775 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.178626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38128)
0927 13:48:28.178850 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.178880 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.179023 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.179405 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.179410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.179879 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.180111 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.180125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.181110 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.181117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.182483 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.182497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.183223 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.183230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.183520 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.184185 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.184214 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.187053 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.190139 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.190144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.190148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.190622 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.190963 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.190968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.190972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.191128 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.191640 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.191645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.191820 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.191988 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.192313 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 13:48:28.193877 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:28.195036 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.194508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37378)
0927 13:48:28.194716 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.194823 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 13:48:28.196228 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.195778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37382)
0927 13:48:28.195968 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.196082 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 13:48:28.196465 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.221541 24491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 3
I20260927 13:48:28.222589 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.222150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37384)
0927 13:48:28.222350 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.222441 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 13:48:28.223613 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.223212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37388)
0927 13:48:28.223422 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.223506 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260927 13:48:28.223809 24491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.247706 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 1
I20260927 13:48:28.248842 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.248330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37398)
0927 13:48:28.248535 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.248636 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:48:28.249922 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.249516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37402)
0927 13:48:28.249680 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.249770 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 13:48:28.250186 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.318415 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 2
I20260927 13:48:28.319631 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.319070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37404)
0927 13:48:28.319324 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.319440 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 13:48:28.320622 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.320248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37414)
0927 13:48:28.320402 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.320491 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:48:28.320828 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
W20260927 13:48:28.340517 24498 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:28.340831 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:28.340947 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:28.341073 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:28.348471 24498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:28.348551 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:28.348713 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:28.348789 24498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:28.348904 24498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:28.359890 24498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:45931,127.1.237.253:44299,127.1.237.252:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.237.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:28.361166 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:28.362438 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:48:28.367024 24491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 4
I20260927 13:48:28.368346 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.367787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37430)
0927 13:48:28.368056 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.368158 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 13:48:28.369436 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.369065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37438)
0927 13:48:28.369217 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.369299 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:48:28.369638 24491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
W20260927 13:48:28.373948 24503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:28.373921 24504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:28.374056 24506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:28.374680 24498 server_base.cc:1034] running on GCE node
I20260927 13:48:28.375392 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:28.376524 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:28.377729 24498 hybrid_clock.cc:648] HybridClock initialized: now 1790516908377678 us; error 21 us; skew 500 ppm
I20260927 13:48:28.377985 24498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:28.380957 24498 init.cc:377] Logged in from keytab as kudu/127.1.237.252@KRBTEST.COM (short username kudu)
I20260927 13:48:28.382191 24498 webserver.cc:460] Webserver started at http://127.1.237.252:46715/ using document root <none> and password file <none>
I20260927 13:48:28.382807 24498 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:28.382925 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:28.383311 24498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:28.385973 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data/instance:
uuid: "679cff2b03c0403790388fe70bf2446f"
format_stamp: "Formatted at 2026-09-27 13:48:28 on dist-test-slave-7g3l"
server_key: "5d46587550f4353b20036657c2a58295"
server_key_iv: "32c0288d3593aba19ffccbcc670ee82d"
server_key_version: "encryptionkey@0"
I20260927 13:48:28.386701 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal/instance:
uuid: "679cff2b03c0403790388fe70bf2446f"
format_stamp: "Formatted at 2026-09-27 13:48:28 on dist-test-slave-7g3l"
server_key: "5d46587550f4353b20036657c2a58295"
server_key_iv: "32c0288d3593aba19ffccbcc670ee82d"
server_key_version: "encryptionkey@0"
I20260927 13:48:28.391389 24498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 13:48:28.394390 24513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:28.395596 24498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 13:48:28.395762 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "679cff2b03c0403790388fe70bf2446f"
format_stamp: "Formatted at 2026-09-27 13:48:28 on dist-test-slave-7g3l"
server_key: "5d46587550f4353b20036657c2a58295"
server_key_iv: "32c0288d3593aba19ffccbcc670ee82d"
server_key_version: "encryptionkey@0"
I20260927 13:48:28.395982 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:28.424019 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:28.427037 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 3
I20260927 13:48:28.427799 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:28.428076 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:28.428110 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.427637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37454)
0927 13:48:28.427832 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.427913 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:48:28.429103 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.428699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37464)
0927 13:48:28.428875 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.428961 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 13:48:28.429356 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:34691: Network error: Client connection negotiation failed: client connection to 127.1.237.252:34691: connect: Connection refused (error 111)
I20260927 13:48:28.445503 24498 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.252:34691
I20260927 13:48:28.445536 24564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.252:34691 every 8 connection(s)
I20260927 13:48:28.446678 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
I20260927 13:48:28.449994 24565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:28.454098 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24498
I20260927 13:48:28.454282 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790516567150638-1975-0/minicluster-data/master-2/wal/instance
I20260927 13:48:28.454689 1975 external_mini_cluster.cc:1428] Setting key 776c725f7ade1f110a294c7de88fa8bf
I20260927 13:48:28.456532 24565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:28.476320 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.458847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.1.237.254:45931)
0927 13:48:28.459014 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.459018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.459798 (+ 780us) server_negotiation.cc:407] Connection header received
0927 13:48:28.460036 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.460041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.460120 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.460223 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.461812 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.462903 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.464953 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.465587 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.465703 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.469560 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.469565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.469568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.469597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.472404 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.473184 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.473189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.473196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.473331 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.473815 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.473819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.473822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.474206 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.474349 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.475786 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.475988 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:48:28.476567 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.458843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:45931 (local address 127.0.0.1:34164)
0927 13:48:28.459665 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.459697 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.459876 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.460305 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.460311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.461142 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.461633 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.461666 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.463046 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.463050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.464805 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.464820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.465796 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.465801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.465976 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.466651 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.466680 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.469372 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.472538 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.472551 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.472561 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.473066 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.473449 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.473456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.473460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.473680 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.474338 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.474344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.474581 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.474848 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.475275 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"spinlock_wait_cycles":12032,"thread_start_us":450,"threads_started":1}
I20260927 13:48:28.476621 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.456304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.1.237.252:34691)
0927 13:48:28.457058 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.457071 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.457107 (+ 36us) server_negotiation.cc:407] Connection header received
0927 13:48:28.459195 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.459228 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.459697 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.460036 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.461717 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.463366 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.465077 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.465875 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.466029 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.470153 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.470160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.470169 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.470195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.472811 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.473553 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.473561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.473569 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.473698 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.474054 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.474061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.474064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.474581 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.474890 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.475192 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.475379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"spinlock_wait_cycles":15744,"thread_start_us":354,"threads_started":1}
I20260927 13:48:28.478341 24565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:48:28.492211 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.478909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:44299 (local address 127.0.0.1:38130)
0927 13:48:28.479042 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.479061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.479149 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.479812 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.479816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.480231 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.480408 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.480420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.481428 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.481436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.482729 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.482743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.483630 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.483638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.483814 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.484329 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.484354 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.486805 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.489877 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.489882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.489887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.490227 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.490584 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.490589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.490593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.490753 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.491295 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.491300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.491448 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.491599 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.491860 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 13:48:28.492247 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.479344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.1.237.253:44299)
0927 13:48:28.479523 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.479532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.479550 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:48:28.479607 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.479611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.479680 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.479783 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.480580 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.481300 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.482884 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.483492 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.483591 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.486988 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.486993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.486996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.487026 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.489756 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.490341 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.490347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.490351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.490478 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.490877 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.490883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.490886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.491136 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.491332 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.491684 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.491885 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:48:28.493054 24565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:28.494418 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.479999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.1.237.254:45931)
0927 13:48:28.480164 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.480171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.480188 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:48:28.480233 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.480237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.480298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.480397 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.481753 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.482875 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.484356 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.484743 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.484831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.488969 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.488980 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.488983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.489010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.491521 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.492142 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.492153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.492156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.492292 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.492816 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.492821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.492824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.493154 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.493417 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.493893 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.494099 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:28.506428 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.493734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.1.237.252:34691)
0927 13:48:28.494031 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.494036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.494177 (+ 141us) server_negotiation.cc:407] Connection header received
0927 13:48:28.494350 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.494353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.494410 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.494496 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.495262 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.496129 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.497155 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.497716 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.497839 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.501299 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.501305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.501309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.501342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.503837 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.504491 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.504499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.504505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.504653 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.505017 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.505024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.505030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.505305 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.505408 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.505893 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.506215 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":203}
I20260927 13:48:28.506496 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.493784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37478)
0927 13:48:28.494115 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.494130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.494238 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.494536 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.494542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.494962 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.495084 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.495092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.496253 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.496260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.497031 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.497041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.497845 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.497849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.498113 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.498667 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.498688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.501151 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.503972 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.503975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.503977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.504373 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.504760 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.504763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.504766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.504908 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.505431 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.505436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.505604 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.505769 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.506094 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":37}
I20260927 13:48:28.510516 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.495863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.1.237.253:44299)
0927 13:48:28.496030 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.496036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.496052 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:48:28.496202 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.496207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.496270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.496361 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.497433 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.498336 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.500166 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.500830 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.500951 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.505140 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.505144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.505147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.505174 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.508225 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.508935 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.508938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.508940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.509057 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.509407 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.509410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.509412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.509635 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.509817 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.510134 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.510276 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:48:28.512303 24565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: Bootstrap starting.
I20260927 13:48:28.514957 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:28.515827 24565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:28.516822 24565 log.cc:824] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:28.519505 24565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: No bootstrap required, opened a new log
I20260927 13:48:28.517519 24392 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34168
I20260927 13:48:28.526028 24565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.526304 24565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679cff2b03c0403790388fe70bf2446f, State: Initialized, Role: FOLLOWER
I20260927 13:48:28.526701 24565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.527602 24571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.527964 24571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.528494 24565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:48:28.537410 24582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:48:28.537523 24582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:48:28.535233 24498 master_runner.cc:186] Error getting master registration for 127.1.237.254:45931: OK, instance_id { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" instance_seqno: 1790516907468509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:48:28.542829 24498 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:48:28.606575 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 4
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:28.623196 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.607423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37480)
0927 13:48:28.607666 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.607705 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.607836 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.608247 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.608252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.608776 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.608992 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.609005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.610444 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.610450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.612077 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.612092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.612855 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.612860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.613077 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.613753 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.613777 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.616669 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.620462 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.620467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.620473 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.620974 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.621352 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.621357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.621360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.621535 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.622209 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.622214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.622379 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.622512 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.622773 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 13:48:28.623400 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.607554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.1.237.252:34691)
0927 13:48:28.607760 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.607766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.607791 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:48:28.607985 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.607990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.608075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.608184 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.609154 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.610266 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.612223 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.612710 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.612807 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.616871 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.616878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.616883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.616921 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.620324 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.621103 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.621109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.621113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.621247 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.621647 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.621652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.621656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.622091 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.622240 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.622653 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.622872 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:48:28.628122 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: Bootstrap starting.
I20260927 13:48:28.632244 24417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:28.633455 24417 log.cc:824] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:28.636154 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: No bootstrap required, opened a new log
I20260927 13:48:28.641249 24417 raft_consensus.cc:348] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.641588 24417 raft_consensus.cc:729] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95fa3369f6c4e1388ee22de6bcd9e84, State: Initialized, Role: FOLLOWER
I20260927 13:48:28.642009 24417 consensus_queue.cc:260] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.642948 24585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.643435 24585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.644220 24417 sys_catalog.cc:564] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:28.644296 24392 raft_consensus.cc:482] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 13:48:28.644505 24392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:28.647759 24392 raft_consensus.cc:504] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.650104 24392 leader_election.cc:288] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [CANDIDATE]: Term 1 election: Requested vote from peers 7ef8e3410acb4b85b16d57fda5fc7e02 (127.1.237.253:44299), 679cff2b03c0403790388fe70bf2446f (127.1.237.252:34691)
I20260927 13:48:28.651109 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "679cff2b03c0403790388fe70bf2446f"
I20260927 13:48:28.651611 24540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:28.654979 24540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95fa3369f6c4e1388ee22de6bcd9e84 in term 1.
I20260927 13:48:28.655597 24368 leader_election.cc:302] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 679cff2b03c0403790388fe70bf2446f, c95fa3369f6c4e1388ee22de6bcd9e84; no voters:
I20260927 13:48:28.656019 24585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:28.667800 24585 raft_consensus.cc:686] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [term 1 LEADER]: Becoming Leader. State: Replica: c95fa3369f6c4e1388ee22de6bcd9e84, State: Running, Role: LEADER
I20260927 13:48:28.668817 24491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } attempt: 5
I20260927 13:48:28.668730 24585 consensus_queue.cc:237] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.677599 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95fa3369f6c4e1388ee22de6bcd9e84. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.678251 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: This master's current role is: LEADER
W20260927 13:48:28.680379 24519 tablet.cc:2369] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 13:48:28 dist-test-slave-7g3l krb5kdc[24337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:28.689352 24597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:28.693032 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.669521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:34691 (local address 127.0.0.1:37488)
0927 13:48:28.669970 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:28.669995 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:28.670142 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:28.671273 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:28.671292 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:28.671966 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:28.672243 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.672259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.674171 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.674179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.676112 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:28.676132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.677701 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.677711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:28.678029 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.678770 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:28.678802 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:28.683845 (+ 5043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:28.689340 (+ 5495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.689349 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.689356 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.690007 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.690558 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:28.690565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:28.690571 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:28.690764 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:28.691663 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:28.691671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:28.691874 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 13:48:28.692042 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:48:28.692456 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260927 13:48:28.700577 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.670412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.1.237.252:34691)
0927 13:48:28.670639 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.670647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.670672 (+ 25us) server_negotiation.cc:407] Connection header received
0927 13:48:28.670749 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.670756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.670896 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.671015 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.672519 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.673958 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.676356 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.677278 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.677440 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.684124 (+ 6684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.684134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.684141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.684192 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.689117 (+ 4925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.690181 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.690190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.690197 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.690403 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.690931 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.690939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.690945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.691406 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.691586 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.693459 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.700040 (+ 6581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 13:48:28.708537 24371 tablet.cc:2369] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:48:28.708953 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: Bootstrap starting.
I20260927 13:48:28.714221 24491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:28.718765 24597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 13:48:28.719916 24491 log.cc:824] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:28.725107 24491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: No bootstrap required, opened a new log
I20260927 13:48:28.753381 24491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.753865 24491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef8e3410acb4b85b16d57fda5fc7e02, State: Initialized, Role: FOLLOWER
I20260927 13:48:28.754514 24491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } }
I20260927 13:48:28.764240 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.693864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.1.237.252:34691)
0927 13:48:28.716065 (+ 22201us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.716072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.726469 (+ 10397us) server_negotiation.cc:407] Connection header received
0927 13:48:28.726739 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.726747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.726882 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.727007 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.737961 (+ 10954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.739401 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.742075 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.742987 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.743109 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.755486 (+ 12377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.755495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.755503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.755558 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.759965 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.761075 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.761084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.761089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.761270 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.761819 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.761827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.761832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.762273 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.762580 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.763350 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.763663 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22038,"thread_start_us":27725,"threads_started":1}
I20260927 13:48:28.770539 24540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [term 1 FOLLOWER]: Refusing update from remote peer c95fa3369f6c4e1388ee22de6bcd9e84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:48:28.771551 24593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:48:28.773398 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.773754 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.783962 24466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02"
I20260927 13:48:28.784268 24465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:28.783950 24491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:28.794412 24465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [term 1 FOLLOWER]: Refusing update from remote peer c95fa3369f6c4e1388ee22de6bcd9e84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:48:28.798939 24466 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95fa3369f6c4e1388ee22de6bcd9e84 in term 1.
I20260927 13:48:28.801270 24593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:48:28.811493 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95fa3369f6c4e1388ee22de6bcd9e84. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.811887 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.813108 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.759912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.1.237.253:44299)
0927 13:48:28.760066 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.760072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.765926 (+ 5854us) server_negotiation.cc:407] Connection header received
0927 13:48:28.766217 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.766225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.766333 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.766414 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.769149 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.770572 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.773931 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.774762 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.774885 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.801649 (+ 26764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.801660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.801665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.801709 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.806309 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.809559 (+ 3250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.809567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.809574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.809761 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.810341 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.810349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.810355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.810814 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.811054 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.812433 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.812694 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:48:28.816218 24571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95fa3369f6c4e1388ee22de6bcd9e84. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.816507 24571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.817286 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:28.693868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.1.237.254:45931)
0927 13:48:28.694047 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:28.694054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:28.757640 (+ 63586us) server_negotiation.cc:407] Connection header received
0927 13:48:28.757819 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:28.757826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:28.757924 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:28.758035 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:28.761919 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.763053 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.771376 (+ 8323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:28.772083 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:28.772176 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:28.785979 (+ 13803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:28.785989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:28.785996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:28.786049 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:28.791077 (+ 5028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.814120 (+ 23043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.814126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.814129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.814345 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:28.814836 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:28.814841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:28.814844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:28.815257 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:28.815550 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:28.816476 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 13:48:28.816807 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:48:28.822010 24571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.822320 24571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.823971 24610 mvcc.cc:205] Tried to move back new op lower bound from 7333957258281988096 to 7333957257916227584. Current Snapshot: MvccSnapshot[applied={T|T < 7333957258281988096 or (T in {7333957258281988096})}]
I20260927 13:48:28.825480 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.825712 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:28.829424 24597 catalog_manager.cc:1347] Generated new cluster ID: 34e7a439f817437fb7b122fc3a632632
I20260927 13:48:28.829530 24597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:28.838284 24424 master_runner.cc:186] Error getting master registration for 127.1.237.254:45931: OK, instance_id { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" instance_seqno: 1790516907468509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:48:28.855003 24424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:48:28.863125 24585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.863741 24585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:28.863978 24597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:28.865779 24597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:28.873693 24632 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 13:48:28.874414 24630 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95fa3369f6c4e1388ee22de6bcd9e84" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 45931 } } peers { permanent_uuid: "7ef8e3410acb4b85b16d57fda5fc7e02" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 44299 } } peers { permanent_uuid: "679cff2b03c0403790388fe70bf2446f" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 34691 } } }
I20260927 13:48:28.874835 24630 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:28.881958 24628 catalog_manager.cc:1259] Loaded cluster ID: 34e7a439f817437fb7b122fc3a632632
I20260927 13:48:28.882088 24628 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: loading cluster ID for follower catalog manager: success
I20260927 13:48:28.885943 24597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c95fa3369f6c4e1388ee22de6bcd9e84: Generated new TSK 0
I20260927 13:48:28.886945 24597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:28.888952 24628 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: acquiring CA information for follower catalog manager: success
W20260927 13:48:28.890259 24628 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260927 13:48:29.055869 24420 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.032968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.1.237.254:45931)
0927 13:48:29.033174 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.033182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.053814 (+ 20632us) server_negotiation.cc:407] Connection header received
0927 13:48:29.054272 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.054279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.054376 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.054448 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.055586 (+ 1138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34172: BlockingRecv error: recv got EOF from 127.0.0.1:34172 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 13:48:29.058785 24568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.039059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.1.237.252:34691)
0927 13:48:29.039256 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.039263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.056514 (+ 17251us) server_negotiation.cc:407] Connection header received
0927 13:48:29.056707 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.056715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.056802 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.056924 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.058548 (+ 1624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37504: BlockingRecv error: recv got EOF from 127.0.0.1:37504 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 13:48:29.060678 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.057224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.1.237.253:44299)
0927 13:48:29.057442 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.057448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.059228 (+ 1780us) server_negotiation.cc:407] Connection header received
0927 13:48:29.059469 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.059474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.059572 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.059665 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.060465 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38154: BlockingRecv error: recv got EOF from 127.0.0.1:38154 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 13:48:29.104804 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.091923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.1.237.253:44299)
0927 13:48:29.092100 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.092105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.102902 (+ 10797us) server_negotiation.cc:407] Connection header received
0927 13:48:29.103360 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.103366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.103493 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.103562 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.104517 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38164: BlockingRecv error: recv got EOF from 127.0.0.1:38164 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 13:48:29.156719 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.143559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.1.237.253:44299)
0927 13:48:29.143761 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.143768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.154746 (+ 10978us) server_negotiation.cc:407] Connection header received
0927 13:48:29.155283 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.155289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.155395 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.155472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.156489 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38180: BlockingRecv error: recv got EOF from 127.0.0.1:38180 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 13:48:29.205641 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.192603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.1.237.253:44299)
0927 13:48:29.192777 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.192784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.203623 (+ 10839us) server_negotiation.cc:407] Connection header received
0927 13:48:29.204127 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.204135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.204251 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.204325 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.205378 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38192: BlockingRecv error: recv got EOF from 127.0.0.1:38192 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 13:48:29.216243 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.213986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.1.237.253:44299)
0927 13:48:29.214205 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.214211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.214786 (+ 575us) server_negotiation.cc:407] Connection header received
0927 13:48:29.214982 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.214992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.215073 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.215205 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.215983 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38202: BlockingRecv error: recv got EOF from 127.0.0.1:38202 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 13:48:29.229588 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.227494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.1.237.253:44299)
0927 13:48:29.227661 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.227668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.228165 (+ 497us) server_negotiation.cc:407] Connection header received
0927 13:48:29.228347 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.228353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.228447 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.228560 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.229369 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38218: BlockingRecv error: recv got EOF from 127.0.0.1:38218 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 13:48:29.251576 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.249290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.1.237.253:44299)
0927 13:48:29.249495 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.249502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.250040 (+ 538us) server_negotiation.cc:407] Connection header received
0927 13:48:29.250212 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.250218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.250327 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.250436 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.251325 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38232: BlockingRecv error: recv got EOF from 127.0.0.1:38232 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 13:48:29.295337 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.292968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.1.237.253:44299)
0927 13:48:29.293163 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.293169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.293712 (+ 543us) server_negotiation.cc:407] Connection header received
0927 13:48:29.293924 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.293930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.294033 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.294153 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.294999 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38246: BlockingRecv error: recv got EOF from 127.0.0.1:38246 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 13:48:29.365475 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.363917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.1.237.253:44299)
0927 13:48:29.364136 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.364141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.364158 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:48:29.364224 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.364227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.364292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.364379 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.365231 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38254: BlockingRecv error: recv got EOF from 127.0.0.1:38254 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 13:48:29.498297 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.496574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.1.237.253:44299)
0927 13:48:29.496798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.496805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.497037 (+ 232us) server_negotiation.cc:407] Connection header received
0927 13:48:29.497192 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.497196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.497264 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.497351 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.498048 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38262: BlockingRecv error: recv got EOF from 127.0.0.1:38262 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:48:29.539337 24582 catalog_manager.cc:1259] Loaded cluster ID: 34e7a439f817437fb7b122fc3a632632
I20260927 13:48:29.539422 24582 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: loading cluster ID for follower catalog manager: success
I20260927 13:48:29.541738 24582 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: acquiring CA information for follower catalog manager: success
I20260927 13:48:29.543555 24582 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 679cff2b03c0403790388fe70bf2446f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:48:29.762276 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:29.760617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.1.237.253:44299)
0927 13:48:29.760825 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:29.760832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:29.760986 (+ 154us) server_negotiation.cc:407] Connection header received
0927 13:48:29.761229 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:29.761233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:29.761301 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:29.761411 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:29.762127 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38268: BlockingRecv error: recv got EOF from 127.0.0.1:38268 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 13:48:29.892827 24628 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ef8e3410acb4b85b16d57fda5fc7e02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:48:30.286365 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:30.280582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.1.237.253:44299)
0927 13:48:30.281121 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:30.281125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:30.281144 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:48:30.281315 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:30.281319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:30.281485 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:30.281587 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:30.281949 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.282613 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.284101 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.284512 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.284618 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:30.285113 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:30.285741 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:30.285866 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:30.285959 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 13:48:30.286024 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260927 13:48:30.297246 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:30.291906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.1.237.252:34691)
0927 13:48:30.292421 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:30.292425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:30.292533 (+ 108us) server_negotiation.cc:407] Connection header received
0927 13:48:30.292695 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:30.292698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:30.292844 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:30.292988 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:30.293360 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.294006 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.295282 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.295664 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.295759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:30.296188 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:30.296667 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:30.296780 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:30.296865 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 13:48:30.296925 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260927 13:48:30.311249 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:30.303467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.1.237.252:34691)
0927 13:48:30.303622 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:30.303627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:30.304609 (+ 982us) server_negotiation.cc:407] Connection header received
0927 13:48:30.304806 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:30.304814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:30.304935 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:30.305048 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:30.305692 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.307040 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.309017 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.309618 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.309735 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:30.310271 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:30.310392 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:30.310479 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:30.310649 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 13:48:30.310723 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 13:48:30.311524 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:30.303315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.1.237.254:45931)
0927 13:48:30.304710 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:30.304717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:30.304735 (+ 18us) server_negotiation.cc:407] Connection header received
0927 13:48:30.304843 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:30.304846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:30.304998 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:30.305877 (+ 879us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:30.306021 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.306667 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.308824 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.309398 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.309515 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:30.310271 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:30.310716 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:30.310833 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:30.311053 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 13:48:30.311172 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1111,"threads_started":1}
I20260927 13:48:30.313445 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:30.304332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.1.237.253:44299)
0927 13:48:30.304537 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:30.304542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:30.307454 (+ 2912us) server_negotiation.cc:407] Connection header received
0927 13:48:30.307630 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:30.307636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:30.307728 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:30.307824 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:30.308494 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.309347 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.311447 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:30.312028 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:30.312100 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:30.312694 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:30.312812 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:30.312910 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:30.313064 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 13:48:30.313126 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 13:48:30.316291 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24350
I20260927 13:48:30.317423 24412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:30.423485 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24350
I20260927 13:48:30.443285 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24424
I20260927 13:48:30.444447 24486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:30.554633 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24424
I20260927 13:48:30.574004 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24498
I20260927 13:48:30.574990 24560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:30.676303 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24498
2026-09-27T13:48:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4908 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 13:48:30 dist-test-slave-7g3l krb5kdc[24718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 13:48:30 dist-test-slave-7g3l krb5kdc[24718](info): set up 2 sockets
Sep 27 13:48:30 dist-test-slave-7g3l krb5kdc[24718](info): commencing operation
krb5kdc: starting...
W20260927 13:48:31.771733 1975 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.015s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/test-user.keytab.
Sep 27 13:48:31 dist-test-slave-7g3l krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T13:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T13:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.237.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
Entry for principal kudu/127.1.237.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab.
I20260927 13:48:31.997943 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41317
--webserver_interface=127.1.237.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 13:48:32.304128 24731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:32.304463 24731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:32.304579 24731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:32.304706 24731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:32.311721 24731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:32.311797 24731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:32.311904 24731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:32.311996 24731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:32.312106 24731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:32.321523 24731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.254.keytab
--principal=kudu/127.1.237.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.254:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.237.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:32.322718 24731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:32.324018 24731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 13:48:32.335350 24736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:32.335354 24737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:32.335446 24739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:32.336544 24731 server_base.cc:1034] running on GCE node
I20260927 13:48:32.337240 24731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:32.338533 24731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:32.339804 24731 hybrid_clock.cc:648] HybridClock initialized: now 1790516912339748 us; error 49 us; skew 500 ppm
I20260927 13:48:32.340116 24731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:32 dist-test-slave-7g3l krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:32.343281 24731 init.cc:377] Logged in from keytab as kudu/127.1.237.254@KRBTEST.COM (short username kudu)
I20260927 13:48:32.344640 24731 webserver.cc:460] Webserver started at http://127.1.237.254:33757/ using document root <none> and password file <none>
I20260927 13:48:32.345290 24731 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:32.345420 24731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:32.345718 24731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:32.348630 24731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data/instance:
uuid: "1ca6f98c64b3464895ad3ef272b25dc2"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2338bb3b835fe0652f2ca0438f57e78d"
server_key_iv: "27eac5d65c4d7ec8abbf5ae1106881ab"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.349391 24731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance:
uuid: "1ca6f98c64b3464895ad3ef272b25dc2"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2338bb3b835fe0652f2ca0438f57e78d"
server_key_iv: "27eac5d65c4d7ec8abbf5ae1106881ab"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.354390 24731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 13:48:32.357614 24746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:32.358924 24731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 13:48:32.359092 24731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal
uuid: "1ca6f98c64b3464895ad3ef272b25dc2"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2338bb3b835fe0652f2ca0438f57e78d"
server_key_iv: "27eac5d65c4d7ec8abbf5ae1106881ab"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.359367 24731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:32.389389 24731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:32.393133 24731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:32.393370 24731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:32.410645 24731 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.254:41317
I20260927 13:48:32.410673 24797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.254:41317 every 8 connection(s)
I20260927 13:48:32.411922 24731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/data/info.pb
I20260927 13:48:32.415261 24798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:32.417364 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24731
I20260927 13:48:32.417543 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-0/wal/instance
I20260927 13:48:32.417982 1975 external_mini_cluster.cc:1428] Setting key 09129111a975ca4f05068a69a57dcda7
I20260927 13:48:32.421248 24798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:32 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.237.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.253@KRBTEST.COM" created.
I20260927 13:48:32.444056 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.424000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41317 (local address 127.0.0.1:46214)
0927 13:48:32.424753 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.424801 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:32.427449 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:32.428263 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:32.428273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:32.429045 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:32.429546 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.429573 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.431035 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.431042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.432882 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.432903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.433717 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.433726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.433967 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.434593 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:32.434631 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:32.437061 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:32.440082 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.440093 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.440103 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.440474 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.440838 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.440845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.440848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.440984 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.441684 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:32.441692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:32.441903 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 13:48:32.442156 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 13:48:32.442750 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":159,"thread_start_us":335,"threads_started":1}
I20260927 13:48:32.444077 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.425183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.1.237.254:41317)
0927 13:48:32.425883 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:32.425892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:32.425941 (+ 49us) server_negotiation.cc:407] Connection header received
0927 13:48:32.427785 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:32.427790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:32.428012 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:32.428178 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:32.429751 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.430884 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.433016 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.433496 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.433610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.437264 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:32.437271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:32.437281 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:32.437310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:32.439960 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.440602 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.440608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.440610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.440736 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.441090 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.441095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.441099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.441556 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:32.441756 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:32.442280 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 13:48:32.442643 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
I20260927 13:48:32.446745 24798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:32.447777 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.447416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38176)
0927 13:48:32.447566 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.447673 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 13:48:32.448040 24749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
I20260927 13:48:32.449541 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.449222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38188)
0927 13:48:32.449369 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.449426 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 13:48:32.449774 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:43711: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
Entry for principal kudu/127.1.237.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab.
I20260927 13:48:32.470046 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:43711
--webserver_interface=127.1.237.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:48:32.499141 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } attempt: 1
I20260927 13:48:32.500394 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.499882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38192)
0927 13:48:32.500089 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.500229 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 13:48:32.501317 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.500970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38202)
0927 13:48:32.501101 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.501176 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 13:48:32.501504 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:43711: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
I20260927 13:48:32.569761 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } attempt: 2
I20260927 13:48:32.571053 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.570474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38210)
0927 13:48:32.570731 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.570888 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:48:32.572060 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.571670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38216)
0927 13:48:32.571823 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.571912 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 13:48:32.572232 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:43711: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
I20260927 13:48:32.663493 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } attempt: 3
I20260927 13:48:32.664844 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.664204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38226)
0927 13:48:32.664464 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.664615 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 13:48:32.666004 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.665536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38232)
0927 13:48:32.665717 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.665824 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:48:32.666195 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:43711: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
W20260927 13:48:32.783550 24805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:32.783854 24805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:32.783918 24805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:32.783996 24805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:32.790828 24805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:32.790911 24805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:32.790968 24805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:32.791010 24805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:32.791071 24805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:32.800293 24805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.253.keytab
--principal=kudu/127.1.237.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.253:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.237.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:32.801443 24805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:32.802613 24805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:48:32.809424 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } attempt: 4
I20260927 13:48:32.810566 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.810062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38234)
0927 13:48:32.810354 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.810443 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 13:48:32.811650 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.811284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38244)
0927 13:48:32.811418 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.811503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 13:48:32.811868 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.253:43711: Network error: Client connection negotiation failed: client connection to 127.1.237.253:43711: connect: Connection refused (error 111)
W20260927 13:48:32.813521 24810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:32.813553 24811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:32.814417 24813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:32.814893 24805 server_base.cc:1034] running on GCE node
I20260927 13:48:32.815675 24805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:32.817041 24805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:32.818275 24805 hybrid_clock.cc:648] HybridClock initialized: now 1790516912818208 us; error 42 us; skew 500 ppm
I20260927 13:48:32.818624 24805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:32 dist-test-slave-7g3l krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:32.821729 24805 init.cc:377] Logged in from keytab as kudu/127.1.237.253@KRBTEST.COM (short username kudu)
I20260927 13:48:32.823200 24805 webserver.cc:460] Webserver started at http://127.1.237.253:33743/ using document root <none> and password file <none>
I20260927 13:48:32.823859 24805 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:32.823942 24805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:32.824188 24805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:32.826906 24805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data/instance:
uuid: "994cecce4b5f455cbd408ab9ad285c41"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2737ce018307f2a847fd28b48db2296d"
server_key_iv: "6c259724910ff33d3fea49a717c9ab57"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.827661 24805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal/instance:
uuid: "994cecce4b5f455cbd408ab9ad285c41"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2737ce018307f2a847fd28b48db2296d"
server_key_iv: "6c259724910ff33d3fea49a717c9ab57"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.832516 24805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 13:48:32.835716 24820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:32.836998 24805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:48:32.837131 24805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal
uuid: "994cecce4b5f455cbd408ab9ad285c41"
format_stamp: "Formatted at 2026-09-27 13:48:32 on dist-test-slave-7g3l"
server_key: "2737ce018307f2a847fd28b48db2296d"
server_key_iv: "6c259724910ff33d3fea49a717c9ab57"
server_key_version: "encryptionkey@0"
I20260927 13:48:32.837293 24805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:32.854079 24805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:32.857722 24805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:32.857940 24805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:32.875696 24805 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.253:43711
I20260927 13:48:32.875716 24871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.253:43711 every 8 connection(s)
I20260927 13:48:32.876957 24805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/data/info.pb
I20260927 13:48:32.880381 24872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:32.880757 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24805
I20260927 13:48:32.880957 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-1/wal/instance
I20260927 13:48:32.881417 1975 external_mini_cluster.cc:1428] Setting key 0d1de42ba92dd8826dd7029ea7980347
I20260927 13:48:32.886593 24872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:32 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.237.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.237.252@KRBTEST.COM" created.
I20260927 13:48:32.909973 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.889256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.1.237.254:41317)
0927 13:48:32.889434 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:32.889439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:32.890641 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 13:48:32.893513 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:32.893518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:32.893586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:32.893675 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:32.895655 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.896293 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.898750 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.899199 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.899303 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.903392 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:32.903402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:32.903408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:32.903459 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:32.907074 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.907949 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.907953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.907955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.908073 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.908437 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.908440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.908442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.908721 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:32.908835 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:32.909562 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 13:48:32.909728 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 13:48:32.911435 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.889721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41317 (local address 127.0.0.1:46216)
0927 13:48:32.890455 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.890500 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:32.893213 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:32.893933 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:32.893957 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:32.894719 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:32.895482 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.895510 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.896461 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.896469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.898577 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.898594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.899430 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.899440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.899747 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.900440 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:32.900472 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:32.903108 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:32.907262 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.907268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.907280 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.907687 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.908165 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.908168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.908170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.908326 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.908813 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:32.908819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:32.909150 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 13:48:32.909418 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 13:48:32.909937 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260927 13:48:32.913132 24872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:32 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:32.929484 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.913906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38248)
0927 13:48:32.914087 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.914114 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:32.914231 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:32.915226 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:32.915233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:32.915686 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:32.915893 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.915907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.917810 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.917818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.919096 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:32.919108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.920415 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.920419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:32.920553 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.921115 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:32.921131 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:32.923059 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:32.927251 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.927254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.927257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.927589 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.927987 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:32.927990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:32.927992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:32.928120 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:32.928832 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:32.928836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:32.928937 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 13:48:32.929041 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 13:48:32.929218 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260927 13:48:32.930008 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.913924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.1.237.253:43711)
0927 13:48:32.914501 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:32.914510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:32.914540 (+ 30us) server_negotiation.cc:407] Connection header received
0927 13:48:32.914591 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:32.914601 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:32.914815 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:32.914957 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:32.916072 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.917661 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.919324 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:32.920262 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:32.920402 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:32.923311 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:32.923326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:32.923346 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:32.923388 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:32.927109 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.927715 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.927726 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.927732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.927879 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:32.928219 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:32.928226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:32.928232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:32.928715 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:32.928927 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:32.929220 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 13:48:32.929486 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":395,"thread_start_us":228,"threads_started":1}
I20260927 13:48:32.932530 24872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:32.933516 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.933100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41496)
0927 13:48:32.933292 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.933386 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 13:48:32.933732 24823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:32.935267 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.934890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41506)
0927 13:48:32.935042 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.935112 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 13:48:32.935489 24872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
Entry for principal kudu/127.1.237.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab.
I20260927 13:48:32.943886 1975 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskijlVCN/build/asan/bin/kudu
/tmp/dist-test-taskijlVCN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:45045
--webserver_interface=127.1.237.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.237.212:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 13:48:32.984798 24872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 1
I20260927 13:48:32.986083 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.985478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41518)
0927 13:48:32.985721 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.985853 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 13:48:32.987217 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:32.986770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41534)
0927 13:48:32.986948 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:32.987042 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 13:48:32.987439 24872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.055684 24872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 2
I20260927 13:48:33.056885 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.056369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41540)
0927 13:48:33.056583 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.056692 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 13:48:33.057909 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.057534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41542)
0927 13:48:33.057683 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.057763 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 13:48:33.058099 24872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.111129 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } attempt: 5
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:33.125499 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.112122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.1.237.253:43711)
0927 13:48:33.112326 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.112333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.112355 (+ 22us) server_negotiation.cc:407] Connection header received
0927 13:48:33.112452 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.112457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.112518 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.112599 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.113535 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.114188 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.115814 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.116231 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.116417 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.120242 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.120247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.120250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.120280 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.122987 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.123740 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.123745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.123747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.123860 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.124295 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.124299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.124301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.124584 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.124710 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.125121 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.125275 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:48:33.125636 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.111870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38264)
0927 13:48:33.112063 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.112084 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.112213 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.112677 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.112681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.113158 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.113392 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.113404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.114326 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.114331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.115661 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.115679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.116452 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.116461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.116693 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.117278 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.117299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.120045 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.123117 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.123122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.123125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.123634 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.123969 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.123974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.123977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.124136 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.124718 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.124723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.124882 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.125019 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.125281 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 13:48:33.126834 24798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:33.128033 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.127521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41550)
0927 13:48:33.127721 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.127830 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 13:48:33.129102 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.128633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41556)
0927 13:48:33.128848 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.128954 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 13:48:33.129376 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.149345 24872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 3
I20260927 13:48:33.150456 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.149937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41560)
0927 13:48:33.150146 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.150263 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 13:48:33.151657 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.151122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41568)
0927 13:48:33.151390 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.151488 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260927 13:48:33.151933 24872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.180639 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 1
I20260927 13:48:33.181813 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.181288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41574)
0927 13:48:33.181512 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.181621 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 13:48:33.182796 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.182437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41584)
0927 13:48:33.182580 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.182664 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 13:48:33.183094 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.251353 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 2
I20260927 13:48:33.252517 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.252003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41588)
0927 13:48:33.252222 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.252324 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 13:48:33.253499 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.253135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41598)
0927 13:48:33.253288 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.253377 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:48:33.253715 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
W20260927 13:48:33.266070 24879 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 13:48:33.266420 24879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 13:48:33.266538 24879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 13:48:33.266677 24879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 13:48:33.273645 24879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 13:48:33.273747 24879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 13:48:33.273883 24879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 13:48:33.273964 24879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 13:48:33.274050 24879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 13:48:33.283118 24879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.237.212:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.237.254:41317,127.1.237.253:43711,127.1.237.252:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/krb5kdc/kudu_127.1.237.252.keytab
--principal=kudu/127.1.237.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.237.252:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.237.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 10:07:22 UTC on ccycloud.quasar-eukpsk.root.comops.site
build id 4675733
ASAN enabled
I20260927 13:48:33.284373 24879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 13:48:33.285629 24879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 13:48:33.295179 24872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 4
I20260927 13:48:33.296478 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.295839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41610)
0927 13:48:33.296157 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.296274 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 13:48:33.296897 24885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:33.296872 24887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 13:48:33.297664 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.297217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41622)
0927 13:48:33.297437 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.297527 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 13:48:33.297636 24884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 13:48:33.297901 24872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.298758 24879 server_base.cc:1034] running on GCE node
I20260927 13:48:33.299536 24879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 13:48:33.300727 24879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 13:48:33.301931 24879 hybrid_clock.cc:648] HybridClock initialized: now 1790516913301883 us; error 23 us; skew 500 ppm
I20260927 13:48:33.302189 24879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516913, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 13:48:33.305231 24879 init.cc:377] Logged in from keytab as kudu/127.1.237.252@KRBTEST.COM (short username kudu)
I20260927 13:48:33.306509 24879 webserver.cc:460] Webserver started at http://127.1.237.252:39951/ using document root <none> and password file <none>
I20260927 13:48:33.307102 24879 fs_manager.cc:359] Metadata directory not provided
I20260927 13:48:33.307240 24879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 13:48:33.307538 24879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 13:48:33.310231 24879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data/instance:
uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93"
format_stamp: "Formatted at 2026-09-27 13:48:33 on dist-test-slave-7g3l"
server_key: "72832334acaa5d7991193af4ebf69911"
server_key_iv: "45b7dfaac0696b8a02754e88bc7c279d"
server_key_version: "encryptionkey@0"
I20260927 13:48:33.310915 24879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal/instance:
uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93"
format_stamp: "Formatted at 2026-09-27 13:48:33 on dist-test-slave-7g3l"
server_key: "72832334acaa5d7991193af4ebf69911"
server_key_iv: "45b7dfaac0696b8a02754e88bc7c279d"
server_key_version: "encryptionkey@0"
I20260927 13:48:33.315577 24879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 13:48:33.318778 24894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 13:48:33.320031 24879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 13:48:33.320168 24879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data,/tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal
uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93"
format_stamp: "Formatted at 2026-09-27 13:48:33 on dist-test-slave-7g3l"
server_key: "72832334acaa5d7991193af4ebf69911"
server_key_iv: "45b7dfaac0696b8a02754e88bc7c279d"
server_key_version: "encryptionkey@0"
I20260927 13:48:33.320320 24879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 13:48:33.343590 24879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 13:48:33.347671 24879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 13:48:33.347872 24879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 13:48:33.359921 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 3
I20260927 13:48:33.360988 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.360565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41632)
0927 13:48:33.360754 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.360834 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 13:48:33.362005 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.361675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41646)
0927 13:48:33.361832 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.361912 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 13:48:33.362286 24798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.237.252:45045: Network error: Client connection negotiation failed: client connection to 127.1.237.252:45045: connect: Connection refused (error 111)
I20260927 13:48:33.365206 24879 rpc_server.cc:307] RPC server started. Bound to: 127.1.237.252:45045
I20260927 13:48:33.365226 24945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.237.252:45045 every 8 connection(s)
I20260927 13:48:33.366505 24879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/data/info.pb
I20260927 13:48:33.369763 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 13:48:33.376148 1975 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskijlVCN/build/asan/bin/kudu as pid 24879
I20260927 13:48:33.376330 1975 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskijlVCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790516567150638-1975-0/minicluster-data/master-2/wal/instance
I20260927 13:48:33.375787 24946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:33.376839 1975 external_mini_cluster.cc:1428] Setting key 58a9091e86807753bb3310dec1dcb33b
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516913, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:33.397475 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.378961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.1.237.254:41317)
0927 13:48:33.379145 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.379176 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.379891 (+ 715us) server_negotiation.cc:407] Connection header received
0927 13:48:33.381416 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.381421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.381499 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.381610 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.383359 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.384457 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.386447 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.387093 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.387283 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.391033 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.391039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.391043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.391081 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.394336 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.395201 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.395207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.395213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.395366 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.395862 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.395867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.395870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.396170 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.396312 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.396946 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.397144 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:48:33.398299 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.378870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.1.237.252:45045)
0927 13:48:33.379620 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.379634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.379673 (+ 39us) server_negotiation.cc:407] Connection header received
0927 13:48:33.381177 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.381197 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.381475 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.381656 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.383401 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.384903 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.386568 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.387428 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.387539 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.391820 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.391831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.391847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.391889 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.395100 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.395711 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.395718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.395722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.395870 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.396261 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.396267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.396270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.396611 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.396927 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.397212 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.397447 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":543,"thread_start_us":402,"threads_started":1}
I20260927 13:48:33.398317 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.378896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.254:41317 (local address 127.0.0.1:46224)
0927 13:48:33.379745 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.379781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.381190 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.381696 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.381703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.382579 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.383101 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.383146 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.384618 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.384622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.386298 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.386314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.387337 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.387341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.387543 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.388187 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.388218 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.390871 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.394462 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.394471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.394477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.395063 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.395476 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.395481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.395485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.395729 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.396304 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.396311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.396598 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.396855 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.397258 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":2,"thread_start_us":442,"threads_started":1}
I20260927 13:48:33.399816 24946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516913, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.254@KRBTEST.COM
I20260927 13:48:33.412608 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.400558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.253:43711 (local address 127.0.0.1:38276)
0927 13:48:33.400718 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.400735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.400819 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.401106 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.401110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.401527 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.401710 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.401720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.402587 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.402592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.403618 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.403631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.404258 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.404263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.404470 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.405028 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.405043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.406955 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.410059 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.410063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.410067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.410509 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.410951 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.410956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.410959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.411131 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.411727 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.411732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.411897 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.412061 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.412298 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 13:48:33.412664 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.400580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.1.237.253:43711)
0927 13:48:33.400790 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.400794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.400810 (+ 16us) server_negotiation.cc:407] Connection header received
0927 13:48:33.400918 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.400922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.400976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.401082 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.401829 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.402453 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.403750 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.404128 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.404211 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.407125 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.407129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.407132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.407200 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.409931 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.410649 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.410655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.410659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.410825 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.411256 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.411260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.411263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.411589 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.411748 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.412144 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.412327 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 13:48:33.413558 24946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } has no permanent_uuid. Determining permanent_uuid...
I20260927 13:48:33.415306 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.401197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.1.237.254:41317)
0927 13:48:33.401340 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.401344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.401399 (+ 55us) server_negotiation.cc:407] Connection header received
0927 13:48:33.401557 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.401560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.401603 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.401672 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.402699 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.403399 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.404838 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.405293 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.405386 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.409172 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.409176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.409179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.409205 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.412285 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.413183 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.413188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.413192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.413365 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.413734 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.413740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.413743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.414023 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.414278 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.414769 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.414959 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516913, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.252@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.237.253@KRBTEST.COM
I20260927 13:48:33.427655 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.414113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:41676)
0927 13:48:33.414252 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.414268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.414368 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.414874 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.414880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.415264 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.415420 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.415431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.416853 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.416865 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.417844 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.417861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.418562 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.418567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.418784 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.419672 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.419691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.421536 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.425268 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.425273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.425277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.425748 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.426146 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.426153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.426156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.426319 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.426774 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.426791 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.426952 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.427058 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.427317 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 13:48:33.427874 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.414441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.1.237.252:45045)
0927 13:48:33.414547 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.414551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.414570 (+ 19us) server_negotiation.cc:407] Connection header received
0927 13:48:33.414597 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.414600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.414665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.414742 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.415524 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.416410 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.417976 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.418416 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.418549 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.421704 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.421708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.421711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.421736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.425118 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.425886 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.425892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.425897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.426022 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.426436 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.426439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.426441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.426656 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.426754 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.427204 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.427511 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":27}
I20260927 13:48:33.432063 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.416760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.1.237.253:43711)
0927 13:48:33.416883 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.416888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.417050 (+ 162us) server_negotiation.cc:407] Connection header received
0927 13:48:33.417150 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.417155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.417219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.417310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.418725 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.419796 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.421278 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.421839 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.421922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.426888 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.426892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.426895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.426920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.429712 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.430437 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.430440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.430442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.430539 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.430910 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.430912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.430914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.431144 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.431390 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.431660 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.431800 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 13:48:33.433027 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: Bootstrap starting.
I20260927 13:48:33.436996 24946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:33.438444 24946 log.cc:824] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:33.439347 1975 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 13:48:33.441723 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: No bootstrap required, opened a new log
I20260927 13:48:33.442512 24773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ca6f98c64b3464895ad3ef272b25dc2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46238
I20260927 13:48:33.448678 24946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.449004 24946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea4f17bf3dd4ba2b3be44f9df7a3a93, State: Initialized, Role: FOLLOWER
I20260927 13:48:33.449436 24946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.450397 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.450822 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.451382 24946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 13:48:33.460196 24963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 13:48:33.460368 24963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 13:48:33.457877 24879 master_runner.cc:186] Error getting master registration for 127.1.237.254:41317: OK, instance_id { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" instance_seqno: 1790516912405764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:48:33.468731 24879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:48:33.539522 24798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 4
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.254@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:33.555028 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.540321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.1.237.252:45045)
0927 13:48:33.540592 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.540599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.540623 (+ 24us) server_negotiation.cc:407] Connection header received
0927 13:48:33.540757 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.540762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.540835 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.540934 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.541832 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.542943 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.544750 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.545439 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.545560 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.549049 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.549055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.549059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.549101 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.552296 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.552909 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.552916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.552919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.553060 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.553435 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.553441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.553444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.553889 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.554046 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.554461 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.554654 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 13:48:33.555027 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.540270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:50782)
0927 13:48:33.540477 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.540498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.540627 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.540973 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.540978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.541486 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.541691 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.541703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.543091 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.543097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.544616 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.544632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.545594 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.545601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.545805 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.546451 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.546474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.548886 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.552436 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.552441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.552445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.552793 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.553208 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.553213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.553216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.553336 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.554043 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.554048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.554190 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.554358 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.554656 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 13:48:33.561088 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: Bootstrap starting.
I20260927 13:48:33.565145 24798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:33.566409 24798 log.cc:824] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:33.569685 24798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: No bootstrap required, opened a new log
I20260927 13:48:33.576995 24798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.577397 24798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca6f98c64b3464895ad3ef272b25dc2, State: Initialized, Role: FOLLOWER
I20260927 13:48:33.577880 24798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.578859 24966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.579253 24966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.580030 24798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:33.580155 24773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 13:48:33.580356 24773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:33.582957 24773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.584124 24773 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 994cecce4b5f455cbd408ab9ad285c41 (127.1.237.253:43711), 6ea4f17bf3dd4ba2b3be44f9df7a3a93 (127.1.237.252:45045)
I20260927 13:48:33.585685 24921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93"
I20260927 13:48:33.586223 24921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:33.590857 24921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca6f98c64b3464895ad3ef272b25dc2 in term 1.
I20260927 13:48:33.591777 24749 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca6f98c64b3464895ad3ef272b25dc2, 6ea4f17bf3dd4ba2b3be44f9df7a3a93; no voters:
I20260927 13:48:33.592308 24966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 13:48:33.594492 24966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca6f98c64b3464895ad3ef272b25dc2, State: Running, Role: LEADER
I20260927 13:48:33.595135 24966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.597146 24872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } attempt: 5
W20260927 13:48:33.599601 24900 tablet.cc:2369] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:48:33.604650 24973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca6f98c64b3464895ad3ef272b25dc2. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.605086 24973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: This master's current role is: LEADER
Sep 27 13:48:33 dist-test-slave-7g3l krb5kdc[24718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790516912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.237.253@KRBTEST.COM for kudu/127.1.237.252@KRBTEST.COM
I20260927 13:48:33.614466 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.597681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.237.252:45045 (local address 127.0.0.1:50784)
0927 13:48:33.598013 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 13:48:33.598033 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 13:48:33.598138 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 13:48:33.598982 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 13:48:33.598988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 13:48:33.599510 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 13:48:33.599730 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.599748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.601150 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.601155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.602546 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 13:48:33.602561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.603732 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.603738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 13:48:33.603956 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.604628 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 13:48:33.604652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 13:48:33.607497 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 13:48:33.611706 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.611711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.611716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.612217 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.612663 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 13:48:33.612668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 13:48:33.612672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 13:48:33.612830 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 13:48:33.613481 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 13:48:33.613486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 13:48:33.613635 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 13:48:33.613793 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 13:48:33.614080 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260927 13:48:33.614748 24977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 13:48:33.615396 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.598350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.1.237.252:45045)
0927 13:48:33.598526 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.598535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.598552 (+ 17us) server_negotiation.cc:407] Connection header received
0927 13:48:33.598610 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.598615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.598709 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.598796 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.599916 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.600990 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.602710 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.603378 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.603504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.607711 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.607719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.607728 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.607784 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.611528 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.612369 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.612374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.612378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.612537 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.612952 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.612958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.612961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.613277 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.613422 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.614804 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.615014 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 13:48:33.621637 24872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: Bootstrap starting.
I20260927 13:48:33.629215 24872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: Neither blocks nor log segments found. Creating new log.
I20260927 13:48:33.631598 24872 log.cc:824] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: Log is configured to *not* fsync() on all Append() calls
I20260927 13:48:33.638094 24977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 13:48:33.645918 24752 tablet.cc:2369] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 13:48:33.649490 24872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: No bootstrap required, opened a new log
I20260927 13:48:33.654592 24921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [term 1 FOLLOWER]: Refusing update from remote peer 1ca6f98c64b3464895ad3ef272b25dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:48:33.655684 24973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:48:33.672308 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca6f98c64b3464895ad3ef272b25dc2. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.672603 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.680887 24986 mvcc.cc:205] Tried to move back new op lower bound from 7333957278317699072 to 7333957278091923456. Current Snapshot: MvccSnapshot[applied={T|T < 7333957278317699072}]
I20260927 13:48:33.681973 24988 mvcc.cc:205] Tried to move back new op lower bound from 7333957278317699072 to 7333957278091923456. Current Snapshot: MvccSnapshot[applied={T|T < 7333957278317699072}]
I20260927 13:48:33.693192 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.637391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.1.237.254:41317)
0927 13:48:33.637571 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.637577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.652674 (+ 15097us) server_negotiation.cc:407] Connection header received
0927 13:48:33.653605 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.653612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.653687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.653758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.659380 (+ 5622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.660872 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.667819 (+ 6947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.668635 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.668737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.674876 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.674883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.674887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.674926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.678886 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.680381 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.680388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.680391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.681231 (+ 840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.691122 (+ 9891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.691130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.691133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.691533 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.691838 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.692461 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.692705 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 13:48:33.696215 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.696466 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.697289 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.651484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.1.237.252:45045)
0927 13:48:33.651657 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.651664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.657361 (+ 5697us) server_negotiation.cc:407] Connection header received
0927 13:48:33.657728 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.657734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.657827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.657893 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.663938 (+ 6045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.665004 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.670596 (+ 5592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.671320 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.671429 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.685533 (+ 14104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.685542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.685549 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.685591 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.689316 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.692123 (+ 2807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.692131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.692136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.692266 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.695088 (+ 2822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.695093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.695097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.695477 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.695743 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.696649 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.696897 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 13:48:33.672468 24731 master_runner.cc:186] Error getting master registration for 127.1.237.253:43711: OK, instance_id { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" instance_seqno: 1790516912870640 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 13:48:33.699702 24731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 13:48:33.702693 24966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.702977 24966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:33.704586 24977 catalog_manager.cc:1347] Generated new cluster ID: db04c9a964734c509eea330b42623de0
I20260927 13:48:33.704685 24977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 13:48:33.718964 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:33.637880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.1.237.253:43711)
0927 13:48:33.638045 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.638051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.650267 (+ 12216us) server_negotiation.cc:407] Connection header received
0927 13:48:33.651500 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.651506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.651587 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.651660 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.681386 (+ 29726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.682437 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.690167 (+ 7730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:33.690837 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:33.690930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:33.700900 (+ 9970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 13:48:33.700907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 13:48:33.700912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 13:48:33.700954 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 13:48:33.705187 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.715829 (+ 10642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.715835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.715839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.716002 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 13:48:33.716739 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 13:48:33.716760 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 13:48:33.716765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 13:48:33.717121 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 13:48:33.717395 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:33.718361 (+ 966us) server_negotiation.cc:299] Negotiation successful
0927 13:48:33.718597 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 13:48:33.719231 24872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.719621 24872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994cecce4b5f455cbd408ab9ad285c41, State: Initialized, Role: FOLLOWER
I20260927 13:48:33.720129 24872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } }
I20260927 13:48:33.728219 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.728597 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.730480 24872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 13:48:33.732139 24847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "994cecce4b5f455cbd408ab9ad285c41"
I20260927 13:48:33.732595 24847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 13:48:33.739895 24977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 13:48:33.742349 24977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 13:48:33.743180 24847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca6f98c64b3464895ad3ef272b25dc2 in term 1.
I20260927 13:48:33.743947 24846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [term 1 FOLLOWER]: Refusing update from remote peer 1ca6f98c64b3464895ad3ef272b25dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 13:48:33.745580 24966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 13:48:33.755224 24977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2: Generated new TSK 0
I20260927 13:48:33.759007 24977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 13:48:33.824777 25004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.825205 25004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.825577 24996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca6f98c64b3464895ad3ef272b25dc2. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.825832 24996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 13:48:33.825505 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ca6f98c64b3464895ad3ef272b25dc2" member_type: VOTER last_known_addr { host: "127.1.237.254" port: 41317 } } peers { permanent_uuid: "994cecce4b5f455cbd408ab9ad285c41" member_type: VOTER last_known_addr { host: "127.1.237.253" port: 43711 } } peers { permanent_uuid: "6ea4f17bf3dd4ba2b3be44f9df7a3a93" member_type: VOTER last_known_addr { host: "127.1.237.252" port: 45045 } } }
I20260927 13:48:33.825915 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca6f98c64b3464895ad3ef272b25dc2 [sys.catalog]: This master's current role is: LEADER
I20260927 13:48:33.881001 25014 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 13:48:33.892210 1975 mini_kdc.cc:348] Time spent kdestroy: real 0.118s user 0.000s sys 0.025s
W20260927 13:48:33.898888 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.895672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.1.237.253:43711)
0927 13:48:33.895886 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.895892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.896347 (+ 455us) server_negotiation.cc:407] Connection header received
0927 13:48:33.897223 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.897232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.897338 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.897428 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.898613 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38662: BlockingRecv error: recv got EOF from 127.0.0.1:38662 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 13:48:33.901403 24949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.896433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.1.237.252:45045)
0927 13:48:33.896603 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.896608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.899529 (+ 2921us) server_negotiation.cc:407] Connection header received
0927 13:48:33.899807 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.899815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.899923 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.900023 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.901107 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50806: BlockingRecv error: recv got EOF from 127.0.0.1:50806 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 13:48:33.903650 24802 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.898397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.1.237.254:41317)
0927 13:48:33.898607 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.898613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.901885 (+ 3272us) server_negotiation.cc:407] Connection header received
0927 13:48:33.902139 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.902147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.902235 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.902320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.903346 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38932: BlockingRecv error: recv got EOF from 127.0.0.1:38932 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 13:48:33.928381 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.925378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.1.237.253:43711)
0927 13:48:33.925869 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.925875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.926908 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 13:48:33.927176 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.927181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.927315 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.927441 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.928172 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38676: BlockingRecv error: recv got EOF from 127.0.0.1:38676 (error 108)
Metrics: {"server-negotiator.queue_time_us":361}
W20260927 13:48:33.945021 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.939382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.1.237.253:43711)
0927 13:48:33.939540 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.939545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.943645 (+ 4100us) server_negotiation.cc:407] Connection header received
0927 13:48:33.943875 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.943880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.943957 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.944063 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.944853 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38688: BlockingRecv error: recv got EOF from 127.0.0.1:38688 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 13:48:33.952548 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.950440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.1.237.253:43711)
0927 13:48:33.950644 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.950652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.951169 (+ 517us) server_negotiation.cc:407] Connection header received
0927 13:48:33.951418 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.951422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.951499 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.951603 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.952314 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38692: BlockingRecv error: recv got EOF from 127.0.0.1:38692 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:48:33.961575 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.959627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.1.237.253:43711)
0927 13:48:33.959879 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.959887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.960315 (+ 428us) server_negotiation.cc:407] Connection header received
0927 13:48:33.960507 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.960512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.960587 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.960686 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.961347 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38698: BlockingRecv error: recv got EOF from 127.0.0.1:38698 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 13:48:33.975950 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.973950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.1.237.253:43711)
0927 13:48:33.974158 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.974167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.974637 (+ 470us) server_negotiation.cc:407] Connection header received
0927 13:48:33.974848 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.974853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.974934 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.975039 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.975724 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38700: BlockingRecv error: recv got EOF from 127.0.0.1:38700 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 13:48:33.998108 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:33.995863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.1.237.253:43711)
0927 13:48:33.996104 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:33.996113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:33.996641 (+ 528us) server_negotiation.cc:407] Connection header received
0927 13:48:33.996877 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:33.996881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:33.996966 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:33.997078 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:33.997853 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38702: BlockingRecv error: recv got EOF from 127.0.0.1:38702 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 13:48:34.035989 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:34.034038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.1.237.253:43711)
0927 13:48:34.034232 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:34.034239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:34.034602 (+ 363us) server_negotiation.cc:407] Connection header received
0927 13:48:34.034770 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:34.034776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:34.034877 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:34.035003 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:34.035723 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38710: BlockingRecv error: recv got EOF from 127.0.0.1:38710 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 13:48:34.105336 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:34.103282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.1.237.253:43711)
0927 13:48:34.103532 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:34.103538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:34.103912 (+ 374us) server_negotiation.cc:407] Connection header received
0927 13:48:34.104110 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:34.104114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:34.104187 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:34.104274 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:34.105111 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38722: BlockingRecv error: recv got EOF from 127.0.0.1:38722 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 13:48:34.238588 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:34.236662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.1.237.253:43711)
0927 13:48:34.236884 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:34.236891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:34.237159 (+ 268us) server_negotiation.cc:407] Connection header received
0927 13:48:34.237376 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:34.237381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:34.237483 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:34.237569 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:34.238329 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38730: BlockingRecv error: recv got EOF from 127.0.0.1:38730 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 13:48:34.462294 24963 catalog_manager.cc:1259] Loaded cluster ID: db04c9a964734c509eea330b42623de0
I20260927 13:48:34.462430 24963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: loading cluster ID for follower catalog manager: success
I20260927 13:48:34.464998 24963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: acquiring CA information for follower catalog manager: success
I20260927 13:48:34.466588 24963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ea4f17bf3dd4ba2b3be44f9df7a3a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 13:48:34.499536 24876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 13:48:34.497754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.1.237.253:43711)
0927 13:48:34.497956 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:34.497961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:34.498287 (+ 326us) server_negotiation.cc:407] Connection header received
0927 13:48:34.498462 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:34.498466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:34.498538 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:34.498648 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 13:48:34.499371 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38734: BlockingRecv error: recv got EOF from 127.0.0.1:38734 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 13:48:34.885056 25014 catalog_manager.cc:1259] Loaded cluster ID: db04c9a964734c509eea330b42623de0
I20260927 13:48:34.885196 25014 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: loading cluster ID for follower catalog manager: success
I20260927 13:48:34.887773 25014 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: acquiring CA information for follower catalog manager: success
I20260927 13:48:34.889508 25014 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 994cecce4b5f455cbd408ab9ad285c41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 13:48:35.023136 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:35.016980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.1.237.253:43711)
0927 13:48:35.017799 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:35.017805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:35.017846 (+ 41us) server_negotiation.cc:407] Connection header received
0927 13:48:35.017913 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:35.017917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:35.018068 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:35.018230 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:35.018683 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.019572 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.020859 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.021313 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.021451 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:35.021929 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:35.022472 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:35.022636 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:35.022693 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 13:48:35.022770 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":510,"threads_started":1}
I20260927 13:48:35.036124 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:35.028700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.1.237.252:45045)
0927 13:48:35.030730 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:35.030735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:35.030750 (+ 15us) server_negotiation.cc:407] Connection header received
0927 13:48:35.030846 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:35.030849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:35.030987 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:35.031114 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:35.031599 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.032284 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.033983 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.034337 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.034420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:35.034937 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:35.035527 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:35.035631 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:35.035758 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 13:48:35.035820 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1932,"thread_start_us":1813,"threads_started":1}
I20260927 13:48:35.065644 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:35.050800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.1.237.253:43711)
0927 13:48:35.051010 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:35.051019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:35.057538 (+ 6519us) server_negotiation.cc:407] Connection header received
0927 13:48:35.058048 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:35.058057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:35.058175 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:35.058250 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:35.059142 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.060624 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.063277 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.063874 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.063970 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:35.064819 (+ 849us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:35.064941 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:35.065007 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:35.065243 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 13:48:35.065333 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 13:48:35.071251 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:35.050470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.1.237.254:41317)
0927 13:48:35.063058 (+ 12588us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:35.063063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:35.063499 (+ 436us) server_negotiation.cc:407] Connection header received
0927 13:48:35.063626 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:35.063633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:35.063909 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:35.064754 (+ 845us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:35.064957 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.065969 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.068047 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.068664 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.068882 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:35.070042 (+ 1160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:35.070494 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:35.070557 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:35.070753 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 13:48:35.070893 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12463,"thread_start_us":12594,"threads_started":1}
I20260927 13:48:35.072351 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 13:48:35.063201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.1.237.252:45045)
0927 13:48:35.064135 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 13:48:35.064139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 13:48:35.066306 (+ 2167us) server_negotiation.cc:407] Connection header received
0927 13:48:35.066559 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 13:48:35.066566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 13:48:35.066653 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 13:48:35.066714 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 13:48:35.067285 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.068297 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.070422 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 13:48:35.070988 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 13:48:35.071078 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 13:48:35.071654 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 13:48:35.071784 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 13:48:35.071896 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 13:48:35.072026 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 13:48:35.072110 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817}
I20260927 13:48:35.077123 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24731
I20260927 13:48:35.078104 24793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:35.208240 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24731
I20260927 13:48:35.227967 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24805
I20260927 13:48:35.229056 24867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:35.337159 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24805
I20260927 13:48:35.357131 1975 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskijlVCN/build/asan/bin/kudu pid 24879
I20260927 13:48:35.358139 24941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 13:48:35.464766 1975 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskijlVCN/build/asan/bin/kudu with pid 24879
2026-09-27T13:48:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4786 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9695 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (348274 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST